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Preface 


At  the  request  of  the  Advisory  Group  for  Aerospace  Research  and  Development  (AGARD)  Flight  Mechanics  Panel,  the 
author  attempted  to  prepare  a  document  outlining  the  rudiments  of  reliability  and  maintainability  (R&M)  evaluations 
conducted  during  initial  flight  test  programs.  Many  military  organizations  prefer  to  defer  R&M  evaluations  until  the  new 
equipment  has  been  delivered  to  the  eventual  user.  Other  organizations  do  not  structure  R&M  engineering  as  an  integral  part 
of  the  flight  test  team.  The  U.S.  Air  Force  Flight  Test  Center  at  Edwards  AFB,  California  has  long  conducted  R&M 
evaluations  during  initial  flight  test  and  this  document  is  written  from  that  perspective. 

The  AGARDograph  presumes  an  entry  level  R&M  engineering  skill  and  does  not  dwell  on  R&M  fundamentals. 

It  is  hoped  that  this  AGARDograph  will  satisfy  any  need  for  understanding  of  R&M  evaluations  conducted  during  initial 
flight  test. 


Preface 


A  la  demande  du  Panel  AGARD  de  la  Mecanique  du  vol,  I’auteur  a  redige  un  document  qui  pre.sente  les  principes  de  base  des 
evaluations  de  fiabilite  et  de  maintenabilite  (R&M)  effcctuces  lors  des  programmes  des  premiers  essais  en  vol.  Bon  nombre 
d’organisations  militaires  choisissent  de  differer  les  evaluations  R&M  jusqu’a  ce  que  le  matiSriel  neuf  ait  ete  r&eptionn6  par 
I’utilisateur.  D’autres  organisations  prel^rent  ne  pas  integrer  I’ingenierie  R&M  dans  les  fonctions  de  I’equipe  d’essais  en  vol. 
Depuis  tr6s  longtemps,  le  centre  d’essais  en  vol  de  I’US  Air  Force  a  Edwards,  en  Californie,  realise  des  evaluations  R&M 
lors  des  premiers  essais  en  vol  et  ce  document  est  redig<5  dans  cetle  optique. 

La  leeture  de  cette  AGARDographie  exige,  toutefois,  un  certain  niveau  de  connaissances  en  ing6nierie  R&M. 

Ce  document  doit  permettre  une  meilleure  comprehension  des  evaluations  R&M  effcctuees  lors  des  premiers  essais  en  vol. 
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RELIABILITY  AND  MAINTAINABILITY  FLIGHT  TEST  TECHNIQUES 


by  Jan  M.  Howell 
412  Test  Wing/DOER 
Edwards  Air  Force  Base,  California,  USA 


SUMMARY 


Reliability  and  maintainability  (R&M)  evaluations  can 
be  conducted  during  the  initial  flight  test  of  new  and 
modified  aerospace  systems.  Newly  developed 
equipment  usually  has  only  a  fraction  of  the  reliabihty 
needed.  These  evaluations,  combined  with  extensive 
laboratory  test  efforts,  are  required  to  bring  the  system 
to  an  acceptable  level  of  R&M  performance.  No  flight 
time  is  normally  dedicated  to  these  R&M  evaluations, 
but  some  ground  time  is  required.  Other  unique 
required  resources  include  trained  R&M  engineering 
personnel,  R&M  data,  and  data  reduction  capability. 
Results  include  identification  of  R&M  deficiencies  and 
measurement  of  R&M  parameters. 

1.0  INTRODUCTION 

1.1  Purpose  of  Volume 

This  AGARDograph  provides  information  to  the 
reader  who  must  evaluate  reliability  and 


maintainability  (R&M)  of  aeronautical  weapons 
systems  during  initial  flight  test.  Fhght  test  R&M 
evaluations  are  essential  because  R&M  characteristics 
cannot  be  predicted  with  any  degree  of  success.  Initial 
reliability  for  newly  designed  equipment  is  normally 
10  to  20  percent  of  the  predicted  value.  The  actual 
reliabihty  begins  to  approach  the  original  prediction 
only  after  considerable  laboratory  and  flight  test. 

Table  1  lists  the  predicted  reliability,  laboratory  test 
rehability,  and  the  fleet  use  reliabihty  for  several 
different  types  of  U.S.  aeronautical  equipment.  The 
large  differences  between  predictions,  laboratory  test, 
and  actual  use  vividly  show  the  need  for  indepth  flight 
test  R&M  evaluations.  This  volume  concentrates  on 
fhght  test  because  of  the  demonstrated  need  to  evaluate 
systems  during  actual  usage.  Laboratory  testing  is 
discussed  only  as  it  relates  to  flight  test. 


Table  1 


Specified,  Predicted,  Demonstrated,  and  Actual  Reliability 
Meantime  Between  Failures  in  Hours 
(Data  from  a  1987  Rome  Air  Development  Center,  U.S.  Air  Force  Study) 


EQUIPMENT 

CONTRACT 

SPECIFIED 

CONTRACTOR 

PREDICTED 

LABORATORY 

TEST 

ACTUAL  USE 

ALQ-131 

(F-4G  radar  warning 

17 

59 

47 

268 

ALQ-135 

(F-15C  radar  warning) 

131 

169 

231 

66 

APG-66 

(F-16  attack  radar) 

80 

150 

55 

97 

APQ-114 

(FB-111  attack  radar) 

137 

185 

212 

22 

ARC-164 
(F-lllUHF  radio) 

1,000 

1,626 

374 

168 

ARC- 164 

(B-52H  UHF  radio) 

1,000 

1,626 

374 

843 
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R&M  evaluations  consist  of  two  general  areas:  R&M 
engineering  and  R&M  accounting.  R&M  engineering 
is  the  engineering  practice  needed  to  yield  a  reliable 
and  maintainable  product.  This  is  the  science  (or 
perhaps  art)  of  designing  and  manufacturing 
equipment  suitable  for  the  intended  operational  and 
support  enviroranents.  During  flight  test  R&M 
engineering  is  the  process  of  finding  and  fixing 
problems.  R&M  accounting,  in  contrast,  measures  how 
well  a  particular  equipment  suits  a  specific  operational 
and  support  environment.  This  volume  addresses  both 
the  engineering  and  accounting  aspects  of  R&M 
evaluations. 

1.2  Scope 

This  volume  will  address  the  "how  to"  of  conducting 
flight  test  R&M  evaluations  of  aeronautical  systems. 
These  techniques  have  been  used  to  test  aircraft, 
missiles,  and  munitions.  While  these  methods  are 
somewhat  general  in  nature  the  reader  must  modify 
these  concepts  for  use  in  nonairbome  environments. 
For  example,  this  volume  uses  flight  hours  as  a  measure 
of  operational  use.  A  ground-based  cargo 
transportation  system  might  use  torme-kilometers  as  a 
measure  of  operational  usage.  Any  such  needed 
re-interpretations  are  left  to  the  concerned  reader. 

This  volume  also  presumes  that  a  reader  has  knowledge 
of  the  fundamental  tenets  of  reliability  and 
maintainability.  Many  excellent  textbooks,  for  the 
novice  and  for  the  sophisticate,  are  readily  available. 

This  volume  addresses  system  level  evaluations.  That 
is,  evaluations  are  not  hmited  to  the  air  vehicle  but 
include  aspects  of  the  system  such  as  ground  support 
equipment,  facilities,  and  trainers.  Further,  test  articles 
such  as  airborne  avionics  and  subsystem  test  beds 
provide  much  useful  information.  Many  ground  test 
facihties  such  as  anechoic  chambers  and  avionic 
integration  laboratories  are  co-located  with  and  used 
during  flight  test.  The  R&M  data  from  these  facilities 
should  be  used  in  conjunction  with  data  from  flight  test. 

Most  of  this  volume  discusses  evaluations  suitable  for 
systems  with  a  possibility  of  production  in  significant 
quantity.  Those  systems  or  air  vehicles  built  purely  for 
research  purposes  (such  as  the  American  X-29)  require 
different  treatment  because  vehicle  availability  and 
cost  of  ownership  are  secondary  considerations  for 
such  limited  life  efforts. 


1.3  Organization 

The  first  major  section  of  this  volume  discusses  the 
objectives  of  an  R&M  evaluation  in  some  detail.  This 
level  of  knowledge  is  needed  to  advocate,  plan,  and 
conduct  R&M  evaluations.  The  remainder  of  the 
volume  proceeds  in  same  order  as  a  development 
program  would.  First,  the  development  and  acquisition 
process  and  the  R&M  flight  test  engineer’s  role  is 
discussed.  When  the  evaluation  objectives  and 
acquisition  process  are  understood,  the  groundwork  is 
laid  for  a  discussion  of  the  planning  and  preparation 
needed  for  a  successful  evaliration. 

Next,  test  conduct,  data  analysis,  and  results  reporting 
are  discussed.  At  this  point,  the  flight  test  effort  is 
complete.  But,  the  test  community  must  monitor  the 
fleet  usage  of  the  aircraft  to  learn  of  any  test  oversights 
and  a  chapter  of  the  volume  addresses  that  process. 

Finally,  the  volume  presents  some  future  R&M 
considerations.  This  section,  perhaps  optimistically, 
lists  some  evolving  R&M  engineering  tools  and 
technologies  that  will  lead  to  improvement  in 
aerospace  systems. 

1.4  Acknowledgments 

The  use  of  specific  references  has  been  deliberately 
minimized.  This  was  done  to  make  the  volume 
applicable  to  a  variety  of  aeronautical  systems.  This 
approach  also  lowers  the  possibility  of  inadvertently 
including  any  proprietary  or  sensitive  informatioa 

Much  of  any  value  arising  from  this  document 
originated  with  those  long-departed  individuals  who 
taught  and  mentored  the  author  throughout  a  career. 
The  remaining  value  was  added  by  individuals  from  the 
North  Atlantic  Treaty  Organization  (NATO)  nations 
who  generously  contributed  time  and  energy  to  consult 
with  the  author  and  review  the  document.  Weaknesses 
and  errors  belong  to  the  author. 

2.0  R&M  TEST  OBJECTIVES 

2.1  Introduction 

"When  you  don’t  know  where  you  are  going,  any  road 
will  get  you  there."  Before  starting  any  endeavor,  first 
understand  the  objectives  and  benefits.  The  remainder 
of  this  chapter  will  discuss  the  objectives  of  an  R&M 
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evaluation.  Potential  benefits,  along  with  some 
examples,  will  also  be  showa 

2.2  Reliability  Maturation 

The  most  important  objective  of  any  R&M  evaluation 
is  to  increase  system  reliability,  lower  life  cycle  cost, 
and  increase  mission  capability.  Initial  versions  of  new 
hardware  usually  have  10  to  20  percent  of  the  sought 
after  rehability;  the  ground  and  flight  reliability  test 
effort  is  really  a  reliability  maturation  program.  A 
common  misconception  is  that  R&M  evaluations  are 
only  to  measure  R&M  values.  Measurement  for 
measurement’s  sake  can  be  a  sterile  exercise;  the  real 
value  is  in  getting  the  information  needed  to  improve 
the  product.  This  point  caimot  be  over  emphasized. 
Dechning  defense  budgets  dictates  military  utihty  must 
be  maximized  by  weapons  systems  that  work.  In  the 
world  envirorunent  today,  quahty  is  viewed  as  the  most 
important  characteristic  of  any  product.  The  nation,  the 
manufacturer,  and  the  ultimate  consumer  that  do  not 
understand  this  are  destined  for  extinction. 

The  classical  reliability  improvement  effort  is  often 
called  "test-analyze-fix."  In  a  perfect  world,  equipment 
would  work  correctly  when  first  delivered  to  the 
customer.  That  is  not  now,  nor  soon  likely  to  be,  the 
case.  To  engineer  is  a  human  endeavor  and  therefore, 
prone  to  errors.  Such  errors  become  noticeably  more 
frequent  on  the  forefront  of  technology.  And  military 
systems  are  always  required  to  counter  the  most  current 
threat.  That  dictates  the  continual  use  of  leading  edge 
technology.  So,  eliminating  initial  reliability  problems 
will  always  be  a  challenge  in  military  systems. 

The  reliability  improvement  process  is  commonly 
called  "reliabiUty  growth.”  Like  in  nature,  this  growth 
does  not  occur  unless  conditions  are  right.  Maximum 
growth  occurs  when  the  object  is  constantly  fertihzed 
with  money  to  find  and  fix  problems. 

The  process  is  simply  to  identify  the  root  cause  of 
failures  that  occur  during  flight  test  and  eliminate  the 
cause.  Experience  shows  that  the  vast  majority  of 
failures  do  not  require  a  design  change  to  eliminate  the 
failure  cause.  Instead,  changing  the  manufacturing 
process  to  remove  latent  defects  corrects  over  80 
percent  of  the  problems.  Changes  to  the  manufacturing 
process  are  usually  inexpensive  and  do  not  take  long  to 
implement.  A  common  example  from  the  early 
microelectronics  era  was  electrostatic  damage.  One 
memorable  case  required  hundreds  of  engineering 


man-hours  to  find  the  exact  failure  cause.  The  problem 
was  eventually  traced  to  the  work  of  one  technician 
whose  job  was  to  insert  microelectronics  packages  into 
circuit  boards.  The  workshop  was  properly  designed  to 
prevent  electrostatic  damage.  However,  when 
observed,  the  technician  was  not  using  all  of  the 
protective  equipment.  When  asked  why  the  employee 
had  not  worn  the  wrist  earthing  strap,  the 
(irjresponsible  assembler  said  that  the  strap  was 
uncomfortable.  A  brief  discussion  about  potential 
employment  discontinuities  caused  the  rehability  of 
the  employee’s  particular  work  to  increase 
dramatically. 

Other  changes  that  do  not  require  a  re-design  include 
changes  of  parts  suppliers  and  increasing  parts  quality. 
These  changes  are  more  expensive  and  require  more 
time.  The  most  expensive  efforts  are  those  where  the 
actual  equipment  design  must  be  changed.  A  common 
cause  of  re-design  are  initial  designs  that  are  not  suited 
to  the  operating  environment.  One  U.S.  attack  aircraft 
cockpit  display  failed  during  every  gunfire  mission, 
hardly  a  desirable  feature  in  an  air-to-ground  weapons 
system. 

Figure  1  shows  the  percentage  of  environmentally 
caused  failures  by  specific  environmental  factor. 
Temperature  or  vibration  cause  most  environmental 
problems  and  may  require  instramenting  the  aircraft  to 
define  the  problem.  The  instrumentation  installation, 
data  acquisition  and  analysis,  and  eventual  re-design 
are  long  and  costly  processes.  One  recent  U.S.  avionic 
development  program  was  blessed  with  both  thermal 
andvibrationproblems  sufficient  to  cause  an  18-month 
program  delay. 

2.3  Maintainability  Maturation 

The  idea  of  "maintainability  growth"  is  not  as  accepted 
or  as  well  studied  as  reliability  growth,  but 
maintainability  will  improve  if  enough  resources  are 
correctly  applied  to  that  objective.  While  the 
improvement  will  not  be  an  order  of  magnitude  (as 
sometimes  happens  in  reliability),  the  resultant  cost 
savings  and  increase  in  capabihty  will  be  worthwhile. 
The  two  segments  of  a  repair  task  most  amiable  to 
improvement  are  fault  isolation  and  fault  correction. 
Improved  trouble-shooting  procedures,  test 
equipment,  and  built-in-test  capability  will  all  decrease 
task  time.  Special  training  will  improve  performance 
of  very  difficult  tasks. 


TEMP 
40  % 


Figure  1  Environmental  Effects  on  Reliability 
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A  rapidly  developing  technology  for  fault  isolation  is 
the  use  of  the  so-called  "expert  system"  from  the  field 
of  artificial  intelligence.  These  systems  queiy  the  user 
about  the  failure  symptoms,  apply  these  symptoms  to 
an  internal  set  of  mles,  and  derive  a  diagnosis. 
Development  of  these  rules  is  a  natural  flight  test 
objective. 

The  fault  correction  segment  of  repair  time  is  largely 
fixed  by  the  physical  design  of  equipment,  but  changes 
to  procedures,  special  tools,  and  training  can  decrease 
the  actual  repair  time.  Sometimes  the  task  may  be 
difficult  enough  to  warrant  design  changes.  A  case  in 
point  was  an  air  superiority  fighter  with  a  high- 
visibility  canopy  that  was  very  hard  to  install  and 
adjust.  The  manufacturer  initiated  and  paid  for  an 
extensive  re-design  in  the  interest  of  improving 
producibility.  In  another  instance,  the  first  attempt  to 
change  an  engine  in  a  flight  test  aircraft  required  six 
clock  hours  to  complete.  After  some  minor  engine 
trailer  changes,  improved  tools  and  training,  the  change 
required  45  minutes. 

A  unique  flight  test  maintainability  objective  is 
development  and  demonstration  of  integrated  combat 
turn  (ICT)  or  "quick-turn"  procedures.  An  ICT  is  the 
recovery  after  landing,  rearming/refueling  and 
re-launch  of  a  combat  aircraft.  Careful  optimization  of 
personnel,  procedures,  and  equipment  placement  is 
necessaiy  to  minimize  time  and  maintain  safety 
standards. 

An  often  overlooked  maintainability  objective  is 
development  or  refinement  of  the  maintenance  plan. 
The  maintenance  plan  defines,  among  other  things, 
whether  a  specific  part  is  repairable  or  discarded  upon 
failure.  For  repairable  parts  the  maintenance  plan  states 
if  a  part  is  repaired  on  the  vehicle,  sent  to  a  local 
specialized  shop,  or  returned  to  a  remote  repair  depot. 
Initial  maintenance  plans  are  based  on  predicted  failure 
frequencies  and  estimated  repair  times.  When  actual 
R&M  data  becomes  available,  the  plans  invariably 
require  considerable  revision.  With  8,000  major 
replaceable  units,  the  B-IB  aircraft  is  an  excellent 
example  of  the  possible  complexity  of  the  maintenance 
planning  task. 

2.4  Duty  Cycle  Improvement 

A  high  pay-back  evaluation  objective  is  to  assure  that 
all  equipment  has  the  lowest  possible  duty  cycle.  That 


is,  to  make  certain  that  vehicle  subsystems  are  operated 
or  stressed  only  when  needed.  While  this  is  a  seemingly 
obvious  objective,  ejqserience  shows  that  almost  all 
aircraft  have  some  equipments  that  operate  more  than 
required.  The  fighter  aircraft  UHF  radio  that  operated 
any  time  that  ground  power  was  applied  to  the  aircraft 
was  a  good  example.  The  only  way  to  prevent  operation 
was  to  open  circuit  breakers  that  were  not  accessible 
from  the  cockpit.  The  aircraft  was  modified  to  prevent 
urmeeded  ground  operation  and  the  radio  reliability 
(measured  inflight  hours)  increased  threefold. 

A  unique  example  occurred  on  a  test  aircraft  that  had  a 
very  high  failure  rate  of  fuel  quantity  probes.  The 
problem  was  most  puzzling  because  similar  aircraft  did 
not  have  the  problem.  Investigation  showed  that  the 
aircraft  was  instmmented  with  a  system  that  required 
lengthy  ground  operating  periods  to  calibrate.  The  fuel 
quantity  system  was  also  energized  whenever  ground 
power  was  applied.  The  test  base  had  a  high  bacteria 
count  in  the  jet  fuel  supply.  The  continuous  voltage 
applied  to  the  capacitive  type  fuel  probes  caused  the 
bacteria  to  "electroplate"  between  the  probe  plates  and 
short  out  the  quantity  sensor.  Changed  operating 
procedures  opened  the  fuel  quantity  system  circuit 
breakers  whenever  possible  and  vanquished  the 
problem. 

Table  2  shows  the  subsystem  operating  time-to-flight 
time  ratios  for  an  attack  aircraft.  Some  equipment,  such 
as  the  automatic  direction  finder,  is  not  used  every 
flight  and  has  an  appropriately  low  ratio.  In  contrast, 
the  electrical  power  system  is  used  during  aircraft 
maintenance  and  has  a  higher  ratio.  These  ratios  are 
obtained  by  using  the  clocks  (also  called  elapsed  time 
indicators  [ETIs])  on  individual  units  and  the  aircraft 
flight  time.  When  test  aircraft  do  not  have  time 
indicators  installed  on  the  individual  units  it  is 
necessary  to  add  clocks.  With  these  ratios,  it  is 
straightforward  to  identify  the  high  usage  items  and  see 
if  beneficial  changes  are  possible. 

Some  currently  used  equipment  clocks  are  unreliable 
mechanical  chronographs  that  are  seldom  used  after 
testing  is  complete.  The  use  of  microelectronics 
"history  chips"  promises  increased  reliability  and 
greater  utility.  These  memory  units  could  record  usage, 
on/off  cycles,  and  other  measures  of  cumulative  stress 
such  as  thermal  cycles.  Other  information  such  as  a 
failure  history  may  prove  of  value. 
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Table  2 

Selected  Equipment  Operating  Versus  Flight 
Time  Ratio  For  the  A-7D  Fighter  Aircraft 
(AFFTC-TR-70-27  A-7D 
_ Category  II  R&M  Evaluation) _ 


EQUIPMENT 

OPERATING  HOUR/ 
FLIGHT  HOUR 

Flight  Controls 

1.3 

Propulsion 

1.3 

Air  Conditioning 

1.3 

Electrical  Power 

2.7 

Lighting 

2.7 

Hydraulics 

1.4 

Automatic  Direction  Finder 

0.4 

Forward  Looking  Radar 

1.3 

Air  Data  Systems 

2.0 

2.5  Contractor  Performance  Verification 


the  measurement  of  contractor  performance  becomes  a 
very  visible  flight  test  objective. 

Similarly,  when  the  contractor  must  prove  minimum 
R&M  performance  levels  during  test  or  correct 
deficiencies  at  his  expense,  the  performance 
measurement  objective  becomes  very  visible. 

2.6  Deficiency  Identification 

A  major  result  of  an  R&M  valuation  is  the 
identification  of  problem  areas  where  corrective  action 
must  be  taken  before  the  system  is  produced  in 
quantity.  Whether  the  problem  is  reliability  or 
maintainability  centered,  the  process  is  similar.  Once  a 
problem  is  suspected  to  exist,  enough  evidence  must  be 
gathered  to  prove  or  disprove  the  problem.  The 
evidence  may  be  instrumentation  data,  film  or  video 
footage,  or  subjective  information  such  as  pilot 
descriptions.  In  many  w^s  the  R&M  test  engineer 
must  build  a  case  for  the  deficiency  just  as  a  lawyer 
prepares  a  case.  The  evidence  must  be  clear  and 
convincing  and  the  seriousness  of  the  problem  must  be 
apparent.  Further,  enough  data  must  be  available  to 
allow  the  manufacturer  to  correct  the  problem. 

2.7  Tmprovements 


The  R&M  performance  requirements  should  be 
included  in  every  aircraft  contract  just  as  other 
requirements  such  as  payload,  range  and  weight  are. 
The  contract  should  also  clearly  state  how  achievement 
of  R&M  requirements  is  to  be  verified.  Because  R&M 
performance  improves  during  the  development  phase 
it  is  not  possible  to  demonstrate  fully  mature  R&M 
characteristics  during  test.  But  it  is  possible  to 
demonstrate  that  satisfactoiy  progress  is  being  made 
towards  achieving  mature  R&M  values.  Aircraft 
contracts  should  state  what  levels  of  R&M 
performance  will  be  achieved  by  completion  of  test. 
Then,  before  a  commitment  to  high-volume  production 
is  made,  needed  fixes  and  the  associated  risks  should 
be  assessed  to  insure  a  high  probability  that  production 
systems  will  have  the  required  R&M  characteristics. 

Many  current  aircraft  contracts  include  financial 
incentives  to  help  insure  satisfactoiy  (or  better)  R&M 
performance.  These  incentives  take  the  form  of 
monetary  awards  (as  much  as  U.S.  $50,000,000)  given 
in  increments  at  major  program  milestones.  When  a 
large  increment  is  to  be  decided  by  flight  test  results, 


During  flight  test  programs  there  are  often 
opportunities  for  substantial  system  improvement  even 
if  the  aircraft  is  not  deficient  in  any  respect.  If  the 
system  works  as  designed  and  as  agreed-to  between  the 
contractor  and  customer,  no  deficiency  exists  but  the 
system  may  not  be  optimal.  This  may  occur  when  a 
major  technology  advance  becomes  available  after 
system  design  but  before  the  beginning  of  full-scale 
production.  Sometimes  contractors  are  reluctant  to  use 
new  techniques  or  technologies  if  they  were  not  the 
originators  (the  not-invented-here  attitude).  An 
obvious  function  of  flight  test  is  to  identify  such 
potential  improvements.  Again  the  evidence  must  be 
clear  and  convincing  and  the  potential  benefit  must  be 
apparent.  Aircraft  radial  tires  offer  an  excellent 
example  of  such  improvements.  While  the  European 
aerospace  community  was  beginning  to  incorporate 
these  improved-life  tires,  some  of  the  American 
manufacturing  community  was  lagging  behind. 
Because  of  flight  test  involvement,  American  fighters 
are  beginning  to  be  equipped  with  radial  tires.  The 
relatively  new  maintenance-free,  sealed  lead-acid 
battery  is  another  example.  When  a  fighter 
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development  program  encountered  problems  with  a 
more  conventional  nickel-cadmium  battery,  the  flight 
test  reliability  engineers  convinced  the  manufacturer  to 
adopt  the  new  battery  and  improve  reUability  while 
significantly  lowering  life  cycle  cost. 

2.8  Mature  System  Capability 

Prediction  or  estimation  of  the  R&M  driven 
capabilities  of  the  weapons  system  is  a  very  valuable 
result  of  the  flight  test  program.  Specifically, 
maximum  sortie  rate  and  aircraft  turn-around  time 
(time  between  sorties)  are  of  interest.  The  problem  of 
predicting  matirre  system  maximum  sortie  rate  in  an 
operational  envirorunent  from  flight  test  data  are 
nontrivial.  A  simulation  model  of  some  complexity 
must  be  used  to  translate  flight  test  measures  such  as 
repair/service  times  and  frequencies  into  operationally 
oriented  measures  such  as  sortie  rate  under  an 
operational  environment. 

In  contrast,  turnaround  time  is  a  fairly  straightforward 
development  and  measurement  process.  The  process 
begins  with  development  of  a  complete  list  of  the  tasks 
to  be  done  during  tirmaroimd.  An  implicit  assumption 
is  that  the  aircraft  does  not  require  repair  and  requires 
only  servicing  before  the  next  flight.  When  the  task  list 
is  complete,  including  persormel  required  and 
equipment  needed,  the  task  sequence  must  be 
optimized  for  minimum  overall  time.  Safety  must  be 
considered  during  the  task-ordering  process.  For 
example,  refuehng  the  aircraft  while  simultaneously 
running  an  ammunition  loader  may  not  be  the  safest 
way  to  order  the  tasks.  Models  or  scale  drawings  are 
helpful  in  determirring  optimal  support  equipment 
placement.  Once  a  plarmed  sequence  is  developed,  it 
(and  perhaps  several  alternate  methods)  should  be 
tested  for  suitability  and  the  overall  time  recorded. 
Normally  such  tests  result  in  refinement  to  planned 
procedmes  and  lower  times. 

2.9  Logistics  Requirements 

Many  mature  system  logistics  requirements  can  be 
readily  predicted  from  flight  test  data.  While  flight  tests 
are  normally  not  long  enough  to  measure  the  reliability 
(and  the  resulting  spares  requirements)  of  all 
components  of  the  aircraft,  those  parts  with  low 
reliabihty  can  be  measured  to  a  reasonable  confidence. 
The  parts  with  the  lowest  reliability  are  usually  the  high 
cost  items  (such  as  airborne  radar)  so  a  large  part  of  the 
spares  budget  can  be  allocated  from  flight  test  data.  The 


useful  life  of  parts  characterized  by  wear-out,  such  as 
tires  and  brakes  can  be  accurately  measured  and  spares 
requirements  forecast. 

Maintenance  persormel  requirements  are  not  so  easily 
determined  because  of  the  large  differences  between 
the  test  environment  and  the  actual  intended  use 
environment.  Once  the  repair/service  frequencies  and 
times  are  measured  in  flight  test,  the  logistics  models 
discussed  earUer  arc  used  to  extrapolate  from  the  test 
environment  to  the  end  use  envirorunent. 

Test  equipment  utilization  rates  are  another  important 
logistics  consideration  that  requires  a  translation  to  be 
meaningful  in  the  operational  environment.  The  flight 
test  task  is  to  measure  the  "shop  visit"  frequency  and 
the  test  equipment  usage  time  for  the  different  parts. 
Next,  a  maximum  acceptable  work  backlog  must  be 
decided.  Because  the  times  that  failures  occur  are 
random,  the  work  backlog  will  also  be  random.  The 
number  of  spares  available  must  be  considered  when 
determining  the  acceptable  work  backlog  for  a  given 
unit.  When  the  backlog  exceeds  the  spares  available, 
an  aircraft  will  be  grounded  until  a  unit  is  repaired. 
When  "visit  rate,"  test  equipment  use  times,  and 
acceptable  backlog  are  known  it  is  an  exercise  in 
queuing  theory  to  estimate  the  number  of  test 
equipment  horns  needed  to  support  a  given  number  of 
aircraft  at  a  given  utilization  rate. 

Program  unique  facihties  (sound  suppressors,  fuel  cell 
repair  bams,  etc.,)  are  treated  very  similarly  to  test 
equipment.  Once  the  "visit"  frequency  is  measured  and 
the  maximum  tolerable  facihty  backlog  is  determined 
then  the  number  or  capacity  of  facilities  is  a  queuing 
theory  problem. 

2.10  Summary 

The  objectives  listed  include;  improving  and 
measuring  R&M  performance,  identifying 
deficiencies,  lowering  usage,  and  estimating  logistics 
support  requirements.  Benefits  include  increased 
in-commission  rates,  lower  support  costs,  and  more 
accurate  logistics  plarming  factors. 

3.0  DEVELOPMENT/ACOUISITION  PROCESS 

3.1  Requirements  Definition 

The  requirements’  definition  process  lays  the 
formdation  for  the  development  of  any  system.  The 
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importance  of  well  defined  and  justified  requirements 
cannot  be  over  emphasized.  Several  studies  show  that 
approximately  75  percent  of  the  system  cost  is  fixed  by 
the  finalized  requirement.  While  the  requirements  may 
change  during  the  process  the  cost  will  not  decrease. 
Generally,  changes  during  the  system  development 
invariably  increase  costs  as  the  system  grows  more 
complex  to  adopt  to  emerging  threats  or  other  changes 
in  the  anticipated  mission. 

The  test  agencies  must  be  involved  in  the  requirements 
definition  process  for  several  reasons.  First,  the 
experienced  tester  has  a  large  repertoire  of  "lessons 
learned."  These  are  both  negative  ("we’ll  never  try  that 
again")  and  positive  ("that  worked  so  good  let’s  try  it 
again  next  time").  This  knowledge  base  can  be 
invaluable  to  the  requirements  formulation  process. 

The  experienced  tester  can  often  add  an  important 
element  of  realism  to  the  process.  When  the 
requirements  definition  is  left  to  the  eventual  system 
user  the  inevitable  result  is  over  specification  or  asking 
for  capabilities  of  marginal  utility  ("goldplating").  An 
excellent  example  was  provided  by  a  trainer  aircraft 
that  originally  required  an  inertial  navigation  system. 
A  simple  calculation  showed  that  the  life  cycle  cost  for 
the  inertial  navigation  system  would  be  billions  of 
dollars  in  total  cost  (approximately  20  percent  of  the 
total  cost  of  ownership). 

3.2  Contractual  Requirements 

Once  relatively  firm  requirements  are  set,  the  next  step 
is  to  translate  requirements  into  contractual  format 
such  as  a  system  specification  (which  states 
requirements)  and  a  statement  of  work  (which  defines 
required  processes).  The  test  agency  must  participate 
in  preparing  the  system  specification  to  ensure  that 
stated  requirements  can  be  measured  during  test.  A 
requirement  without  a  corresponding  compliance 
assessment  is  merely  a  goal.  Further,  because  R&M 
measures  are  very  sensitive  to  the  large  differences 
between  the  test  and  field  use  scenario,  R&M 
contractual  requirements  must  be  carefully  written. 
These  requirements  must  reflect  the  needs  of  the  user, 
but  must  be  measurable  while  the  immature  system  is 
being  tested  in  an  often  beneficial  environment.  There 
are  a  number  of  significant  differences  between  the  test 
and  field  use  scenario.  These  differences  include; 
differences  in  maintainer  skill  levels,  immature 
technical  data,  and  others.  Some  of  these  factors  bias 
R&M  results  pessimistically  while  others  make  the 


system  appear  better  than  it  actually  is.  Again,  the 
experienced  tester  is  often  needed  to  help  state 
requirements  that  realistically  satisfy  the  users,  but  can 
still  be  measured  during  the  test  program. 

The  statement  of  work  defines  processes  to  be  followed 
and  specific  results  to  be  produced  during  the 
contractor’s  development  program.  Ideally,  these 
defined  processes  merely  formalize  good  engineering 
practices  and  ask  that  the  results  be  produced  in  a 
standard  format.  Sometimes  these  defined  processes 
are  a  result  of  painful  experiences  on  previous 
programs  and  represent  a  problem  avoidance 
technique.  Most  of  these  processes  are  incorporated  by 
referring  to  commercial  and  military  standards  rather 
than  having  bulky  statements  of  work  filled  with 
repetitious  detail. 

Flight  test  community  involvement  in  statement  of 
work  preparation  is  dictated  by  the  need  for  contractor 
data  during  the  development  program  including  the 
flight  test  phase. 

3.3  Design  Reviews 

The  design  review  is  a  widely  accepted  program 
management  tool  used  by  the  mihtary  and  commercial 
sectors  during  the  development  process.  The  timing, 
content,  and  conduct  of  such  reviews  are  normally 
detailed  in  the  statement  of  work.  Briefly,  a  design 
review  is  a  meeting  where  the  contractor  presents  the 
technical  and  programmatic  status  of  the  development 
effort.  Before  the  actual  meeting  the  contractor  is  often 
required  to  provide  the  customer  with  a  considerable 
amount  of  technical  material  for  detailed  review. 
Again,  such  material  content  and  delivery  schedule  is 
normally  listed  in  the  statement  of  work.  The  actual 
meeting  then  serves  as  a  fomm  for  the  customer  and 
provider  to  agree  (or  disagree)  on  the  suitability  of  the 
evolving  design. 

The  flight  test  engineering  community  should  be  well 
represented  during  the  design  review  process.  The 
information  presented  at  reviews  is  essential  to  the 
flight  test  planning  effort.  As  the  design  evolves,  the 
flight  test  engineers  must  begin  determining 
instrumentation  requirements,  data  reduction  needs 
and  the  flight  hours  required  to  adequately  test  the 
system. 

Often  the  flight  test  engineers  will  again  serve  as 
a  living  repository  of  lessons  learned  (and  endlessly 


relearned).  A  current  example  originated  with  theB-lB 
aircraft.  As  originally  designed,  the  B-IB  had  many 
warning  tones  built  into  the  system.  These  tones  were 
intended  to  alert  the  crew  of  impending  problems. 
There  were  so  many  that  the  crew  was  unable  to  find 
the  real  fault.  Consequently,  crew  work  load  was 
unnecessarily  increased.  At  a  subsequent  design  review 
a  different  contractor  was  presented  a  system  very 
similar  to  the  disorienting  B  -  IB  warning  scheme.  After 
learning  of  existing  problems,  the  designers  were 
convinced  to  change  from  tones  to  a  voice  warning 
system. 

Often,  flight  test  R&M  engineers  know  what 
equipment  offers  the  best  reliability.  A  major  weapons 
system  was  proposed  to  have  a  10-year  old  design 
TACAN.  The  manufacturer  was  well  pleased  to  learn 
of  a  new  design  that  offered  twice  the  reliability  and 
weighed  25  pounds  less. 

During  the  conduct  of  design  reviews  the  contractor  is 
normally  required  to  present  documents  to  illustrate 
progress  in  developing  the  aircraft.  From  an  R&M 


standpoint,  the  first  of  these  is  the  R&M  program  plan. 
The  plan  is  intended  to  show  how  the  contractor  intends 
to  develop  a  reliable  and  maintainable  system.  With  the 
current  emphasis  on  R&M,  the  R&M  plan  is  often  part 
of  the  data  submitted  for  source  selection.  Following 
source  selection,  the  plan  and  updates  are  discussed  at 
every  design  review. 

While  the  program  plan  is  a  management  document, 
the  Allocations,  Assessments,  and  Analysis  report 
(often  called  the  Triple-A  or  AAA)  is  a  basic  technical 
document.  This  report  is  a  straightforward  result  of  the 
systems  engineering  process  that  allocates  reliability 
and  maintainabihty  requirements  from  the  system  or 
aircraft  level  downward  to  the  subsystems,  and  finally, 
components.  Table  3  is  an  example  of  an  AAA  report. 

Another  systems  engineering  result  is  the  Failure 
Mode,  Effects,  and  Criticality  Analysis  (FMECA). 
This  report  hsts  the  ways  that  the  system  can  fail  and 
the  result  or  impact  on  the  system.  Generally, 
requirements  and  common  sense  state  that  no  single 
failure  should  result  in  the  loss  of  an  aircraft  and  the 


Table  3 

Example  Allocations,  Analysis,  and  Assessment  Report 
(Intermediate  Level  Maintenance  [LRU]  Task  Time  Predictions) 


Line  Replaceable  Unit 

Nomenclature 
Drawing  Number 

Failure  Rate 
(per  milUon  hours) 

Mean  Corrective 
(minutes) 

TPCC 

7564358 

815.08 

25.68 

MBC 

377-6900-100 

248.83 

22.08 

CEU 

717135000 

1,455.98 

27.41 

TPPS 

717138000 

215.78 

14.26 

ECU 

734261 

124.85 

41.36 

FINS 

717212000 

1,328.77 

57.34 

NPPS 

717213000 

158.43 

145.27 

NPCC 

7564358 

1,781.42 

35.68 

ANT/GIM 

2677277 

637.67 

16.75 

XMITTR 

2711321 

557.30 

22.77 

RCVR/EXCTR 

2677331 

545.23 

10.57 

RIU 

2677351 

749.99 

7.64 
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FMECA  is  the  analysis  tool  used  to  demonstrate 
(during  the  design  phase)  that  the  requirement  has  been 
met.  Table  4  is  an  example  of  an  FMECA. 

The  Association  Europeenne  Des  Constructeurs  De 
Material  Aeroepatial  (AECMA)  has  a  standardized 
method  of  collecting  R&M  predictions  from  suppliers 
at  the  time  of  request  for  proposal.  Figure  2  is  a  form 
used  for  providing  rehability  predictions  and  Figure  3 
is  an  analogous  form  for  providing  maintainability 
predictions.  These  type  of  data  are  essential  for 
updating  AAA  reports  and  FMECAs  as  suppliers  are 
selected. 

Both  the  AAA  and  the  FMECA  are  important 
documents  to  the  R&M  engineering  personnel  and 
should  be  thoroughly  studied  before  design  reviews. 
Statements  of  work  must  provide  for  the  reports  to  be 
delivered  in  time  for  careful  engineering  review 
(nominally  30  days).  These  documents  should  be 
examined  for  completeness  appropriate  to  the  stage  of 
the  development  program.  At  the  first  or  early  reviews 
the  contractor  carmot  be  expected  to  have  allocated 
reliabihty  to  the  piece-part  level  nor  considered  all 
possible  failure  modes.  On  the  other  hand,  once  the 
design  is  frozen,  drawings  completed,  and  metal  being 
bent,  the  reliabihty  has  obviously  allocated  by  default 
if  not  by  design. 

Both  the  AAA  and  the  FMECA  should  be  examined 
for  basic  reasonableness  (the  so-called  "sanity  check"). 
If  the  contractor  predicts  far  greater  reliabihty  than 
achieved  for  similar  equipment  the  prediction  must  be 
questioned  to  learn  what  technological  breakthrough 
led  to  the  dramatic  improvement.  A  tragic  case  of 
failure  to  properly  question  reliabihty  predictions 
occurred  with  the  U.  S.  Space  Transportation  System. 
The  original  prediction  for  the  sohd  rocket  boosters 
was  one  catastrophe  failure  in  10,000  uses.  After  the 
Challenger  disaster,  a  historical  review  showed  the  past 
failure  rate  was  1  in  30  for  large  solid  rockets. 

During  the  actual  flight  test  program  the  AAA  and 
FMECA  are  necessary  documents.  As  the  rehability  of 
individual  components  is  measured,  the  results  should 
be  compared  to  contractor  predictions  to  isolate  those 
components  in  need  of  fixes  and  further  development. 

Another  document  that  must  evolve  during  the 
development  process  is  the  Logistics  Support 
Analysis(LSA).  As  shown  in  the  example  in  Table  5, 
the  LSA  hsts,  in  exhaustive  detail,  the  necessary 


information  to  support  the  weapons  system.  The 
extensive  subject  of  logistics  support  can  only  be 
briefly  discussed  here.  Even  so,  the  flight  test  engineer 
must  have  an  imderstanding  of  the  contents  and  value 
of  the  LSA  report.  This  report  should  evolve  as  the 
system  is  designed  and  should  be  the  tool  used  to  make 
the  engineering  tradeoffs  necessary  to  develop  an 
optimal  system  including  the  logistics  "tail." 

Flight  test  engineers  must  stay  current  on  the  evolving 
LSA  to  ensure  that  system-  pecuhar  logistics  features 
such  as  support  equipment  and  faciUties  are  available 
and  tested  concurrently  with  the  air  vehicle.  Further, 
the  LSA  is  the  source  of  repair  level  information  and 
plans  Repair  level  refers  to  what  echelon  of 
maintenance  (on-  aircraft,  shop  or  depot)  is  the  optimal 
level  to  repair  failed  parts.  This  planning  is  also  called 
the  maintenance  concept.  Much  of  this  information  is 
based  on  the  contractor’s  estimates  of  reliabihty  and 
maintainability,  and  as  such,  is  subject  to  change  when 
actual  R&M  performance  data  becomes  available 
during  the  test  program.  As  discussed  previously,  this 
refining  or  "fine  tuning"  is  a  majorbenefitfromthe  test 
program. 

A  classic  example  of  changing  maintenance  plans  is 
from  the  E-3A  Sentry  program.  During  early 
deployment  an  expensive  (U.S.  $31,000)  unit  failed 
because  of  poor  quality  of  a  single  small  part.  The  unit 
was  an  electrical  power  filter  and  had  no  active  parts. 
With  only  passive  parts  such  as  inductors  and 
capacitors,  the  contractor  predicted  an  essentially  zero 
failure  rate.  With  such  a  low  failure  rate,  it  is  not 
economical  to  plan  for  or  purchase  the  equipment 
needed  to  repair  the  filter.  Because  of  the  high  cost,  the 
125-pound  weight  and  the  200  man-hour  replacement 
time,  the  failed  filter  was  not  discarded  but  saved  as  a 
curiosity  item.  Within  the  next  month,  six  more  filters 
failed.  With  U.S.  $217,000  of  throwaway  parts,  it  was 
obviously  time  to  change  the  maintenance  concept. 

During  the  different  design  reviews,  the  contractor 
should  present  data  showing  the  environment  predicted 
for  the  system  components.  As  seen  from  Figure  1  the 
thermal  and  vibration  factors  hold  the  most  interest  for 
R&M  engineers.  Reliability  predictions  in  the  AAA 
report  should  be  based  on  these  predicted 
environments.  When  system  components  are  essential 
for  flight  safety  or  mission  performance,  it  is  wise  to 
instrument  the  system  to  verify  that  the  actual 
temperature  and  vibration  do  not  significantly  exceed 
predictions.  When  the  actual  environment  is  different 
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Supplier  Reliability  Prediction 
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Figure  3  AECMA  Supplier  Maintainability  Prediction 
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Table  5 

Logistics  Support  Analysis 
(U.S.A.  Military  Standard  1388-1A/2A) 


Operations  &  Maintenance  Requirements 

-  Identification  of  hardware,  source,  and 
quantity  required 

-  Frequency  and  duration  of  use 

-  Allocation  of  preventive  and  corrective 
maintenance  needs  between  organizational, 
intermediate  and  depot  levels 

-  Availability  requirements 

Item  Reliability  and  Maintainability  Characteristics 

-  Identification,  source,  and  quantity  required 

-  Availability  requirements 

-  Maintainability  considerations 

-  Function  of  item 

-  Maintenance  concepts  and  qualitative 
maintainability  requirements 

Failure  Mode  and  Effects  Analyses 

-  Failure  modes  and  resulting  effects 

-  Damage  mode  and  resulting  effects 

-  Survivability  and  vulnerability  analysis 

Criticality  and  Maintainability  Analyses 

-  Criticality  analysis 

-  Maintainability  analysis 

-  High  risk  item  identification 

Operation  and  Maintenance  Task  Summary 

-  Identification,  source,  and  quantity  required 

-  Maintenance  task,  level,  time  required 
manning  skills,  support  equipment  needed 

Operation  and  Maintenance  Task  Analysis 

-  Identification,  source,  and  quantity  required 

-  Task  Identification  and  description,  time 
required,  skills 

-  Common/special  tools,  parts,  and  material 
required  for  task 

Personnel  and  support  requirements 

-  Training  requirements,  personnel,  support 
equipment,  and  supply  support  requirements 
per  task 


Support  Equipment  &  Training  Material  Description 

-  Identification,  source  and  quantity  required 

-  Size,  weight,  storage  volume,  and  costs 

-  Functions  to  be  performed 

-  Characteristics  and  installation  factors 

-  Justification  for  new  material/skill 
requirements 

Unit  Tested/Automatic  Test  Program/Training 
Material  Description 

-  Test  program  set  elements 

-  Hardware  and  software  required  for  testing 

Facility  Description 

-  Identification  and  description  of  new  facility 

-  Functions  and  tasks  to  be  performed 

-  Requirements,  design  criteria  lead  times, 
construction,  and  required  utilities 

-  Facility  utilization  rate  and  cost  justification 

Skill  Evaluation  and  Justification 

-  Identification,  source,  and  quantity  required 

-  Skill  specialty  codes 

-  Functions  to  be  performed 

-  Additional  skill  and  training  requirements 

-  Selection  criteria  (physical,  mental  and 
educational) 

Support  Items  Identification 

-  Spare  parts  data 

-  Provision  screening  data 

-  Packaging  data 

Transportability  Engineering 

-  Identification  of  transportability  requirements 
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from  the  expected,  the  predicted  reliability  must  be 
recalculated  and  changes  made. 

3.4  Flight  Readiness  Reviews 

In  the  near  term,  before  the  aircraft’s  first  flight,  it  is 
normal  practice  to  hold  a  series  of  meetings  to  assure 
that  the  vehicle  is  ready  for  flight.  The  contractor 
presents  the  results  from  the  laboratory  and 
qualification  testing  on  subsystems  and  components. 
This  information  is  the  first  real  reliability  data  and 
often  the  first  sign  of  impending  problems.  The  flight 
test  community  should  be  well  aware  of  auy  problems 
encountered  and  the  potential  impact  on  safely  of 
flight  Further,  all  flight  essential  equipment  must  be 
tested  in  some  maimer  and  to  the  largest  extent 
possible.  Any  failures  of  flight  essential  equipment 
must  be  corrected  or  have  acceptable  work-around 
procedures  established.  Mockups,  "iron  birds,"  and 
enviroiunental  test  chambers  are  some  of  the  tools  used 
to  increase  confidence  in  flight  readiness. 

While  reliability  data  from  test  chambers  and  fixtures 
may  not  represent  the  aircraft,  it  is  still  very  useful.  All 
failure  modes  experienced  in  flight  critical  equipment 
must  be  analyzed  and  a  conscious  decision  made  as  to 
the  necessity  of  a  fix. 

4.0  TEST  PLANNING 

4.1  General 

"Well  begun  is  half  done. "  This  is  never  more  true  than 
with  the  test  planning  process.  Initially,  test  planning 
will  be  necessarily  general  in  nature  and  continuously 
refined  as  the  aircraft  approaches  first  flight.  Planning 
must  begin  early  in  the  development  cycle  to  ensure 
that  all  needed  resources  are  available  at  the  beginning 
of  flight  test.  Resources  needed  include  persormel,  test 
assets,  maintenance,  operations,  and  instmmentation 
data,  and  data  reduction  tools. 

4.2  Personnel 

Appropriate  numbers  of  correctly  trained  people  from 
several  backgrounds  are  required  to  conduct  a  flight 
test  R&M  evaluation  Obtaining  and  training  people  is 
a  long-lead  time  process  and  must  begin  early  in  the 
development  cycle.  Ideally,  these  individuals  should 
have  participated  in  the  design  review  process  prior  to 
actual  flight  test.  Such  participation  will  provide 
detailed  knowledge  of  the  aircraft  to  be  tested  and  some 


feeling  for  potential  trouble  areas.  An  excellent 
example  comes  from  the  ground-laimched  cmise 
missile.  During  the  design  review  process,  it  became 
obvious  that  the  gas  turbine  driven  electrical  generators 
used  to  power  the  control  complex  and  launcher  were 
going  to  be  a  reliability  problem.  As  a  result,  work¬ 
around  methods  and  improvements  were  developed 
before  the  beginning  of  test. 

Trained  R&M  engineering  personnel  are  essential  to  a 
flight  test  R&M  evaluation.  The  engineers  should  have 
a  strong  background  in  aircraft  and  aircraft  systems  in 
addition  to  training  and  experience  in  the  principles  and 
practice  of  R&M.  Perhaps  the  ideal  R&M  engineers  are 
those  individuals  that  have  an  in-depth  expertise  in  one 
discipline  and  good  working  knowledge  of  many 
others  involved  in  aircraft  engineering.  The  number  of 
engineering  persormel  needed  is  a  direct  function  of  the 
complexity  of  the  system  being  tested.  For  small 
systems,  such  as  a  primary  trainer  with  minimal 
avionics,  one  engineer  should  suffice.  Complex 
aircraft,  such  as  a  bomber  or  large  cargo  carrier  might 
require  as  many  as  five  engineers.  Table  6  lists  the 
duties  performed  by  these  engineers. 

Table  6 

_ Flight  Test  R&M  Engineer  Duties 

•  Participate  in  preparation  of  requirements 
documentation. 

•  Participate  in  design  and  technical  reviews. 

•  Prepare  test  plans  and  test  information  sheets. 

•  Determine  instrumentation  requirements. 

•  Implement  R&M  data  collection  system  for 
fli^t  test. 

•  Classify  test  R&M  data  as  to  criticality,  etc. 

•  Obtain  failure  analysis  from  contractors  and 
vendors. 

•  Analyze  R&M  data  to  identify  high-failure 
rates  and  high  manhours  consumers. 

•  Monitor  throwaway  parts  to  identify 
deficiencies  with  low  cost  items. 

•  Observe  maintenance  to  identify  deficiencies 
in  fault  isolation,  component  replacement, 
repair  verification,  and  support  equipment. 

•  Verify  effectiveness  of  implemented  design 
changes. 

•  Write  deficiency  reports. 

•  Write  intermediate  and  final  technical 
reports. 

•  Write  lessons  learned  documentation. 
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Staffing  a  test  program  with  maintenance  personnel 
poses  a  dilemma.  From  one  viewpoint  it  is  desirable  to 
have  senior  maintenance  personnel  available  to  gain 
from  their  experience  with  other  aircraft  and  to  have 
these  persoimel  judge  the  suitability  of  maintenance 
issues.  In  contrast,  it  is  necessary  to  determine  how  well 
the  aircraft  can  be  repaired  and  serviced  by  the  average 
maintainer.  An  acceptable  compromise  is  to  have 
junior  people  do  the  actual  work  while  the  senior 
people  observe  and  judge.  In  this  manner,  the 
experienced  people  can  note  the  mistakes,  trials  and 
tribulations  of  the  average  maintainer,  and  develop 
improvements  when  needed.  This  requires  that 
maintenance  personnel  be  assigned  on  a  selective  basis 
and  that  experienced  people  be  made  available  on  a 
continuous  basis  even  though  they  are  not  maintaining, 
but  rather  testing  the  system.  The  actual  number  of 
maintenance  personnel  needed  varies  greatly  with  the 
complexity  of  the  system.  As  a  minimum,  each 
technical  specialty  (such  as  engine  mechanics  and 
avionic  technicians)  should  be  represented  by  at  least 
one  experienced  individual. 

Experienced  flight  crew  personnel  are  also  needed 
during  an  R&M  evaluatioa  When  an  inflight  anomaly 
occurs,  they  must  accurately  report  the  problem  and  the 
situation  of  the  aircraft  when  it  occurred.  The  complete 
observance  and  accurate  reporting  is  essential  to 
finding  and  correcting  problems.  In  many  ways  the 
ability  to  observe  and  report  differentiates  the  test  pilot 
from  the  operational  or  line  pilot.  The  flight  crew  must 
also  assist  in  determining  if  the  anomaly  had  aity 
implications  for  flight  safety  and  if  the  problem 
prevented  completion  of  the  aircraft  mission.  This 
requires  that  the  aircrew  have  operational  experience 
with  the  intended  mission  of  the  system.  It  also  requires 
that  the  aircrew  have  a  detailed  knowledge  of  the  new 
aircraft  and  the  development  effort.  The  best  way  to 
gain  such  knowledge  is  participation  in  design  reviews. 

An  often  neglected  yet  important  system  facet  is  depot 
maintainability.  While  it  is  not  possible  to  take  a  test 
aircraft  to  a  repair  depot  during  flight  test,  it  is  possible 
to  bring  experienced  depot  personnel  to  the  test  site. 
Because  the  depot  has  a  wide  variety  of  technical 
specialties  it  is  usually  most  cost  effective  to  have 
different  specialists  participate  in  flight  test  on  a 
temporary  basis.  They  should  be  temporarily  assigned 
to  the  test  program  long  enough  to  evaluate  their  area 
of  expertise  in  some  depth.  A  good  example  occurred 
during  the  prototype  testing  of  a  tactical  transport.  The 
depot  maintenance  landing  gear  expert  noted  that  the 


type  and  process  used  for  plating  the  main  gear  stmt 
was  very  difficult  to  remove  when  replating  was 
required.  A  more  suitable  plating  was  no  more  costly 
and  the  change  was  made. 

One  final  persormel  consideration  is  assignment 
stability.  It  is  essential  to  keep  the  same  people 
throughout  the  acquisition  process.  It  does  little  good 
to  have  test  persormel  participate  in  the  early  part  of  the 
process,  such  as  design  reviews,  and  then  change  jobs 
prior  to  the  first  flight.  This  is  a  particularly  acute 
problem  in  many  military  organizations  where  periodic 
reassigrunent  is  standard  procedure.  Every  effort  must 
be  made  to  maximize  persormel  continuity.  This  often 
requires  the  liberal  use  of  civilian  personnel  in  key 
positions  when  military  persormel  assignment  stability 
cannot  be  assured. 

4.3  Test  Asset  Requirement 

The  single  most  important  test  program  cost  driver  is 
the  numberofflight  test  hours.  Then  the  single  question 
becomes;  "How  many  flight  hours  are  required  to 
accomplish  an  effective  R&M  evaluation?".  The 
answer  to  that  question  varies  in  direct  proportion  to 
the  complexity  of  the  aircraft  being  tested.  A  simple 
aircraft  such  as  the  U.  S.  Air  Force  T-46  can  be  well 
characterized,  and  a  majority  of  the  R&M  problems 
identified  in  about  700  flight  hours.  In  contrast,  a  large 
aircraft  with  complex  avionics,  such  as  the  B-1  bomber, 
may  require  several  thousand  flight  hours  to  test.  In 
either  case,  it  is  not  cost  effective  to  test  the  aircraft  long 
enough  to  accurately  measure  the  reliability  of  every 
part.  Indeed,  even  avionic  components  with  a 
2,000-hour  meantime  between  failure  (MTBF)  carmot 
be  measirred  with  any  statistical  significance.  But,  an 
avionic  system,  such  as  a  100-  hour  MTBF  radar  can 
be  measirred.  Even  in  the  case  of  the  2,000-hour  MTBF 
component,  the  initial  reliability  will  be  much  lower 
(200  to  400  hours  typically).  The  flight  test  program 
can  measure  the  lower  numbers  and  identify  some  of 
the  corrections  needed  to  achieve  the  desired 
2,000-hour  MTBF.  Considering  the  large  differences 
between  predicted  reliability  and  that  eventually 
achieved,  the  test  program  seldom  affirms  reliabihty, 
but  rather  often  denies  reliability.  That  is  to  say,  test 
results  usually  show  a  much  lower  reliability  than  was 
predicted. 

Normally,  no  flight  test  time  is  dedicated  to  R&M 
evaluations.  Instead,  the  test  program  is  structured 
around  the  flights  required  to  test  the  vehicle  and 
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subsystem  performance  characteristics.  Table  7  shows 
the  number  of  flight  test  hours  for  several  aircraft. 
Then,  as  a  result  of  stressing  the  vehicle  during  test, 
much  failure  and  repair  data  are  available.  There  are 
several  considerations  that  can  maximize  the  resulting 
R&M  data.  First,  all  installed  subsystems  should  be 
operated  every  flight  regardless  if  it  is  needed  for  any 
given  test.  This  will  maximize  the  operating  experience 
onthe  subsystems.  Tofix  ideas,  consider  aircraft  cmise 
performance  testing.  Much  flight  time  is  needed  to 
gather  fuel  consumption  data  throughout  the 
airspeed/altitude  envelope.  During  that  time,  very  little 
of  the  aircraft  avionic  suite  is  essential  to  that  testing. 
However,  the  full  avionic  suite  should  be  operated 
throughout  performance  testing  to  increase  resulting 
R&M  data.  This  operation  must  include  turning  the 
subsystems  on  and  periodically  testing  those 
equipment  during  the  mission.  This  may  be  done  by 
including  the  appropriate  directions  in  the  flight  crew 
checklists  and  flight  cards. 

Table  7 


Fhght  Hours  for  Initial  R&M  Evaluation 


Aircraft 

Hours 

A-7D 

900 

A- 10  A 

1,325 

C-5A 

2,700 

C-141A 

2,500 

F-4E 

700 

F-5A 

845 

F-15A 

2,900 

F-16A 

1,950 

F-16C 

1,345 

Missile  testing  offers  a  similar  opportunity  to  increase 
the  R&M  experience  base.  For  every  test  mission,  the 
carrier  aircraft  should  be  loaded  with  a  full  complement 
of  missiles,  rxrt  just  the  one  missile  needed  for  that 
day’s  test.  In  this  fashion,  the  available  R&M  database 
can  be  increased  substantially.  This  requires  advanced 
planning  in  order  that  a  sufficient  quantity  of  missiles 
is  available  to  the  test  effort.  This  advanced  planning 
and  the  cost  of  early  missile  delivery  is  often  much 
cheaper  than  attempting  to  duplicate  the  flight 


environment  in  some  ground  test  rig  to  gain  the  needed 
reliabihty  data. 

While  little  or  no  flight  time  dedicated  to  R&M 
evaluations,  much  ground  time  is  required  for 
maintainability  demonstrations  and  logistics 
evaluations.  Although  much  maintainabihty 
information  can  be  obtained  from  normal  maintenance, 
most  flight  test  programs  do  not  last  long  enough  for 
all  (or  even  a  significant  sample)  of  maintenance  tasks 
to  arise.  For  this  reason  a  block  of  ground  time  should 
be  set  aside  to  demonstrate  those  interesting  tasks  that 
have  not  naturally  occurred  during  the  test  program. 
The  tasks  of  the  most  interest  are  the  long  duration, 
complex  efforts.  These  maintenance  tasks  might  use 
unique  support  or  test  equipment  that  needed  to  be 
tested  for  functional  adequacy.  This  dedicated  block  of 
ground  time  varies  in  duration  as  a  function  of  the 
aircraft  complexity.  A  simple  primary  trainer  such  as 
the  T-46  should  not  require  more  than  several  weeks 
while  a  vehicle  as  complex  as  the  B-lAbomber  might 
require  months. 

4.4  R&M  Data  Requirements 

The  three  principal  R&M  data  sources  are  flight  crew 
debriefings,  maintenance  records  and  aircraft  special 
instmmentation.  Flight  crew  debriefing  provides  a 
measure  of  usage  (flight  time)  and  a  record  of 
pilot-noted  problems.  Maintenance  records  report  all 
resources  needed  to  maintain  the  vehicle  in 
mission-capable  condition.  The  aircraft  special 
instrumentation  records  assorted  measures  for 
postflight  analysis.  These  three  data  sources  and  the 
required  data  reduction  are  discussed  separately. 

4.5  Flight  Crew  Debriefing 

The  most  readily  obtainable  stress  data  are  aircrew 
debriefing  information.  Because  of  the  relative  ease  of 
use,  many  test  programs  rely  solely  on  this  data  as 
stress  measurement.  Figure  4  shows  an  aircraft 
debriefing  record  used  for  the  F-1 1 1  test  program.  The 
first  line  of  the  form  is  for  identification  information 
such  as:  date,  time,  and  mission  number.  The  second 
line  of  the  form  is  for  time  of  stress  data  such  as  flight 
duration  and  time  in  afterburner.  Also  included  are  data 
about  cyclic  stresses  such  as  wing  sweeps  and  landings. 
The  grided  area  and  corresponding  table  record,  in  a 
rudimentary  fashion,  the  dynamic  pressure  induced 
stresses  seen  by  the  airframe.  Other  cyclic  uses  of 
different  equipments  should  also  be  recorded.  It  should 
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Figxire  4  F-1 1 1  Test  Program  Debriefing  Form 
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be  noted  some  of  that  cyclic  data  may  suffer  in  accuracy 
by  depending  on  aircrew  recording  or  memory.  Speed 
brake  cycles  is  a  good  example.  No  pilot  can  be 
expected  to  accurately  coimt  speed  brake  usage  during 
simulated  air  combat  maneuvers.  In  cases  where 
accuracy  is  important,  instrumentation  should  be  used 
to  record  cychc  use. 

In  addition  to  usage  ,  the  aircrew  should  also  note  aity 
anomalies  that  occur  and  when  they  occurred  in  the 
flight.  This  information  must  be  complete  enough  to 
allow  maintenance  to  diagnose  the  problem  and  to 
allow  R&M  engineers  a  full  understanding  of  the 
possible  failure.  The  aircrew  must  also  record  any 
anomalies  reported  by  the  aircraft  built-in-test  system 
and  note  whether  any  related  symptom  was  observed. 
Newer  automated  recovery  systems  provide  health 
diagnostics  information  for  postflight  analysis.  It  is 
still  a  good  idea  to  debrief  the  aircrew  to  identify  what 
anomalies  were  observed  and  during  which  portions  of 
the  flight  envelope. 

As  the  program  progresses  through  the  design  review 
phases,  the  R&M  test  engineers  should  continuously 
refine  their  requirements  and  plans  for  flightcrew  data. 
Aircrew  debriefing  forms  should  be  developed  jointly 
by  aircrew  and  engineers. 

Debriefing  data  from  aircrews  is  the  simplest 
information  to  convert  to  usable  form.  These  data  are 
normally  a  single  sheet  or  two  per  attempted  sortie.  The 
easiest  way  to  aggregate  and  sutrunarize  these  data  are 
with  the  use  of  desk  top  computers  and  a  commercial 
database  management  system.  The  most  commonly 
used  data  summary  Usts  the  accumulated  stress  (flight 
hours  or  cycles)  per  unit  time  (often  months). 

4.6  Maintenance  Data 

A  considerable  amount  of  information  is  required  from 
the  maintenance  persormel.  Aircraft  maintenance  can 
be  considered  in  two  broad  general  categories; 
scheduled  and  unscheduled.  Scheduled  maintenance  is 
those  maintenance  efforts  whose  need  can  be  foreseen 
and  accomplished  in  a  plarmed  marmer.  Aircraft 
servicing  and  inspections  comprise  the  bulk  of  this  type 
maintenance.  Data  on  scheduled  maintenance  are 
needed  to  measure  the  resource  requirements  for  such 
efforts  and  to  determine  the  time  the  aircraft  is  not 
available  for  "revenue  service"  because  of  the  need  for 
such  maintenance.  Such  scheduled  activities  should  be 
accurately  measured  early  in  the  test  program.  The 


exact  procedures  used  can  be  studied  for  possible 
optimization  to  limit  resource  consumption  and  out  of 
service  time.  When  no  further  optimization  is  possible, 
data  collection  for  scheduled  maintenance  can  be 
discontinued. 

Unscheduled  maintenance  is  that  maintenance  required 
to  restore  the  aircraft  to  operating  condition  after  an 
anomaly.  Data  onunscheduled  maintenance  are  needed 
to  again  measure  resource  requirements,  to  determine 
aircraft  nonavailabihty  and  also  to  determine  the  exact 
cause  of  the  anomaly .  In  addition  to  the  on  aircraft  work 
done,  these  data  must  include  the  "off-aircraft"  work. 
That  is,  all  work  necessary  to  isolate  the  exact  cause  of 
the  anomaly  and  restore  normal  operation  must  be 
included.  During  test  programs  the  acquiring  service 
often  does  not  have  the  capability  to  repair  the  new 
equipment  and  failed  equipment  must  be  returned  to 
the  prime  contractor.  Then  the  prime  contractor  may 
return  the  failed  item  to  a  vendor  or  even  lower  tier 
supplier.  Considerable  planning  is  needed  to  insure 
that,  regardless  of  the  complex  repair  path,  the  needed 
information  is  available  to  the  flight  test  engineers. 

History  shows  that  the  best  way  to  insure  that  the 
needed  data  are  available  is  to  specify  the  requirement 
in  the  original  contract  and  state  that  the  requirement  is 
to  be  levied  on  all  vendors  and  lower  tier  suppliers. 
Some  experienced  contractors  routinely  require 
vendors  and  lower  tier  suppliers  to  provide  repair  data 
and  failure  analysis.  Those  contractors  regard  this 
process  as  simply  good  commercial  practice.  This  is 
not  always  the  case.  The  recent  B-IB  flight  test 
program  was  initially  plagued  with  continuing 
nuisance  hydraulic  leaks  from  a  type  of  coupling  used 
throughout  the  aircraft.  Many  months  and  much 
hydraulic  fluid  passed  while  a  series  of  discussions  was 
required  to  convince  the  contractor  to  take  any  action. 
Eventually  the  contractor  returned  several  leaking 
coupUngs  to  the  coupling  supplier.  Within  days  the 
suppUer  responded  by  acknowledging  responsibility 
forthe  problem  and  stating  what  action  they  would  take 
to  correct  the  defects.  Much  maintenance  time  and 
valuable  flight  test  time  could  have  been  saved  if  the 
prime  contractor  had  an  established  procedure  to  return 
failed  parts,  even  those  considered  notuepairable,  to  the 
original  supplier. 

A  classic  case  involved  a  magnetic  tape  cartridge  used 
to  transfer  mission  data  to  the  aircraft.  During 
laboratory  and  flight  test  over  1 00  of  the  tape  cartridges 
failed  and  the  prime  contractor  did  not  return  ar^  of  the 
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cartridges  to  the  supplier.  The  supplier  only  found  out 
about  the  problem  when  they  questioned  why  the  prime 
contractor  was  ordering  so  maiy  replacement  tape 
cartridges.  Again,  once  the  supplier  became  aware  of 
the  problem  the  defect  was  fixed  in  a  matter  of  days. 

The  information  needed  about  aircraft  maintenance 
varies  somewhat  with  the  individual  program,  but 
generally  the  same  basic  information  is  needed. 
Appendix  A  Usts  the  individual  data  elements  and 
discusses  the  use  of  such  information. 

Historically,  this  data  are  difficult  to  obtain  from 
contractors  after  a  development  contract  has  been 
signed.  The  requirement  for  such  data  must  be  included 
in  the  basic  contract  if  the  contractor  is  to  perform 
maintenance  during  the  flight  test  program.  This 
requirement  was  inadvertently  omitted  from  the  B-IB 
development  contract  and  the  airframe  contractor 
submitted  aU.S.  3. 2-million  dollar  proposal  to  provide 
the  data.  Better  planning  would  have  prevented  that 
problem. 

4.7  Maintenance  Data  Processing 

The  reduction  of  maintenance  data  is  a  much  more 
challenging  task  because  of  the  greater  relative  volume, 
more  sophisticated  database  creation  and  complex  data 
analysis  requirements.  The  volume  of  maintenance 
data  is  such  that  desktop  computers  are  suitable  for  the 
small  test  programs  only.  For  example,  the  F-111 
Digital  Flight  Controls  System  test  program  was 
conducted  using  a  desk  top  computer  to  store  and 
analyze  maintenance  data.  However,  data  collection 
was  limited  to  three  line  replaceable  imits 
(LRUs-commonly  called  "black  boxes").  Further,  the 
test  program  was  only  600  flight  hours  duration. 

Because  desk  top  computers  are  unsuitable,  a  large 
"mainframe"  computer  is  needed  to  store  and  process 
the  maintenance  data  for  large  test  programs.  The  C-5  A 
maintenance  database  was  200  milhon  bits  of 
information  at  the  end  of  the  test  program.  This  also 
requires  fairly  complex  software  to  maintain  and 
analyze  this  amount  of  data. 

The  software  needed  to  create  and  maintain  a 
maintenance  database  varies  depending  on  the 
information  and  format  of  the  raw  data,  but  certain 
general  requirements  exist.  For  example,  as  part  of  the 
database  maintenance  process,  the  times  that  aircraft 
maintenance  started  and  stopped  must  be  converted 


into  man-hours,  active  hours  and  elapsed  hours.  All 
individual  maintenance  actions  (such  as 
troubleshooting,  actual  repair  and  cleanup)  must  be 
linked  together  into  a  single  maintenance  event. 
Further,  all  levels  of  repair  (on-aircraft,  off-aircraft  and 
depot)  must  be  linked  together.  This  complex  linkage 
is  needed  because  the  various  actions  within  a  single 
maintenance  event  often  occur  at  different  times  and 
different  places.  When  all  the  smaller  actions  are 
properly  linked,  the  total  repair  cost,  both  time  and 
material,  are  visible.  Whenproperly  done,  the  database 
should  provide  an  audit  trail  that  begins  with  a 
description  of  the  aircraft  problem  and  concludes  with 
action  taken  to  prevent  recurrence  of  that  problem. 
Table  8  is  a  much  simphfied  example  of  this.  The 
linkage  process  is  the  most  difficult  part  of  R&M  data 
processing. 

Maintenance  data  analysis  computer  reports  vary  from 
the  trivial  to  the  almost  unusably  esoteric.  Generally, 
the  value  of  these  reports  is  inversely  proportional  to 
the  complexity.  A  most  usable  report,  simply  lists  the 
most  frequently  occurring  failures  in  descending  order 
of  frequency.  A  report  of  similar  type  for  the  highest 
maintenance  man-hour  consumers  is  also  of  utility. 
Table  9  summarizes  other  computer  analysis  reports  of 
varying  utility. 

4.8  Failure  Analysis 

Another  type  of  informationthat  must  be  obtained  from 
the  contractor  is  the  detailed  analysis  describing  the 
root  cause  of  failure  (sometimes  called  physics  of 
failure).  This  type  of  analysis  is  essential.  Both 
contractor  and  customer  should  plan  to  perform  such 
analysis  for  every  failure  that  occurs  during  the  flight 
test  program.  This  includes  failures  of  nomepairable 
pieces.  Without  knowledge  of  the  causes  of  failure  it  is 
impossible  to  prevent  reoccurrence.  Again,  this 
irrformation  is  costly  to  obtain  after  the  contract  is 
awarded  and  must  be  included  in  the  earliest  contracts. 
The  prime  contractor  must  be  required  to  levy  this 
requirement  on  all  vendors  and  lower  tier  suppliers. 

4.9  Instrumentation  Data 

For  avionics,  temperature  and  vibration  are  normally 
considered  the  primary  causes  of  failure  with  changing 
levels  of  thermal  and  vibratory  stress  causing  different 
failure  rates.  Table  10  shows  the  MTBF  for  the  U.S. 
Air  Force  standard  TACAN  (AN/ARN- 1 1 8)  in  several 
different  aircraft.  The  greater  than  one  order  of 
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AIRCRAFr:  F-15E  SERIAL:  9100015  FLIGHT  HOURS:  348 

WORK  UNIT  CODE:  74JA0  REPORT#  006501  DATA  CODE:  3 

WHEN  DISCOVERED:  INFLIGHT  HOW  MALFUNCTIONED:  INTERMITTENT 


PROGRAM  DESCRIPTION  VSD  (VERTICAL  SITUATION  DISPLAY  GOES  BLANK  WITH 
AIRCRAFT  VIBRATION  SUCH  AS  SPEED  BRAKES  OR  FLAPS.  RSETS  OK  AFTER  BEING  OFF 
SEVERAL  SECONDS. 

FLIGHT  LINE  MAINTENANCE  TROUBLESHOOT  VSD  SYSTEM.  REMOVE/REPLACE  VSD. 
SYSTEM  OPS  CHECKS  GOOD. 

FAILED  ITEM:  VSD  SERIAL#:  00041611  ETI  METER:  0095 

INSTALLED  ITEM:  VSD  SERIAL#:  00001233  EH  METER:  0085 

SHOP  MAINTENANCE:  REMOVE/REPLACE  X  DEFLECTION  AMPLIFIER  (2A3).  VSD  OPS 
CHECK  GOOD.  AMPLIFIER  NOT  REPAIRABLE  THIS  STATION.  RETURN  TO  VENDOR  FOR  FIX, 
FAILURE  ANALYSIS  AND  CORRECTIVE  ACTION. 

FAILED  ITEM:  2A3  SERIAL#:  00208004  ETI  METER:  0095 

INSTALLED  ITEM:  2 A3  SERIAL#:  00208018  ETI  METER:  0028 

VENDOR  ANALYSIS:  DISCREPANCY  CAUSED  BY  POOR  SOLDERING.  SOLDER  REFLOWED 
AND  UNIT  CHECKS  GOOD.  INVESTIGATION  REVEALED  THAT  SOLDERING  WAS  DONE  BY  AN 
ENGINEER  VICE  SOLDERING  TECHNICIAN.  VENDOR  STATED  SERIAL  NO.  4  WAS  ONLY  UNIT 
HARMED.  ALSO  PROMISED  ENGINEER  WOULD  NOT  REPEAT  ACTION  AND  THAT  NEW  QA 
PROCESS  WOULD  DETECT  ANY  SOLDERING  PROBLEMS.  USAF  INSPECTION  OF  ALL 
DELIVERED  UNITS  FOUND  NO  OTHER  PROBLEMS.  ACTION  CLOSED. 


Table  9 

Computer  Analysis  Reports 


22 


23 


Table  10 

TACAN  (AN-ARN-188)  ReliabiUty  by  Aircraft 
(Data  from  the  U.S.  Air  Force  Logistics  Command 


Databases  of  the  1984  to  86  Time  Period  and  Sample 
Revahdated  for  the  1989  to  90  Time  Period) 


Aircraft 

Meantime  Between  Failure 
(hours') 

F-4G 

199 

FB-lllA 

410 

A-lOA 

774 

F-lllE 

818 

F-16C/D 

3,296 

A-7K 

592 

F-15C 

685 

magnitude  difference  between  the  F4G  and  F-16D 
aircraft  clearly  shows  the  importance  of  the 
environment  in  determining  equipment  reliability. 


Because  of  the  high  cost  of  instrumentation,  Ri&M  test 
engineers  seldom,  if  ever,  get  all  of  the  measurands  that 
they  want.  The  problem  then  becomes  an  allocation 
process.  With  a  limited  capability  to  instrument  the  test 
aircraft,  where  should  instmmentation  transducers  be 
placed  to  maximize  the  value  of  the  information?  The 
problem  becomes  more  acute  when  instrumentation  is 
to  be  installed  during  initial  aircraft  construction.  This 
means  that  the  instrumentation  must  be  specified  in 
parallel  with  the  aircraft  design  effort.  The  obvious  rule 
of  thumb  is  to  instmment  the  most  critical  from  a  safety 
of  flight  standpoint  and  from  a  cost  viewpoint.  Less 
obvious  perhaps  is  the  approach  of  instrumenting  the 
system  to  verify  the  design  predictions.  Many  aircraft 
temperature  predictions  are  made  based  on  results  of  a 
large  computer  model  of  the  system.  Instrumentation 
should  be  designed  to  verify  and  perhaps  improve  the 
model. 

Vibration  sensing  instrumentation  is  more  difficult  to 
plan  The  only  guidelines  are  to  consider  all  vibration 
inducing  sources  and  place  sensors  around  those 
producing  sufficient  energy  to  be  potentially 
troublesome.  Aircraft  mounted  guns  are  always  a  good 
candidate  for  vibration  sensors  because  of  the  very  high 
energy  generated.  Equipment  mounted  close  to  such 
energy  sources  should  be  closely  monitored  for 


vibration.  Engine  and  accessory  power  unit  mounted 
equipment  may  produce  considerable  energy. 

One  lesson  endlessly  relearned  is  the  need  to 
thoroughly  understand  the  vibration  environment  of 
externally  mounted  equipment.  While  external  pods 
often  seem  like  a  good  way  to  enhance  capability  and 
adapt  aircraft  for  special  missions,  the  vibration 
encormtered  can  be  quite  severe.  Other 
nonaerodynamic  additions  to  a  basic  aircraft  also 
induce  similar  problems. 

Predicting  vibration  levels  of  such  externally  carried 
equipment  remains  a  very  inexact  science.  The  only 
current  solution  is  constraction  and  test  of  a  sfructurally 
representative  article  very  early  in  the  development 
cycle.  These  test  articles  must  be  aerodynamically  and 
stracturally  similar  to  the  platmed  equipment.  These 
test  "shapes"  must  have  enough  instmmentation  to 
completely  characterize  the  vibration  environment  at 
the  worst  case  flight  conditions.  Most  importantly, 
these  "shapes"  must  be  tested  on  all  potential  carrier 
aircraft.  In  case  after  case,  missiles  and  external  pods 
are  designed  for  one  aircraft  and  adopted  to  others.  It 
is  somewhat  ironic  that  the  more  successful  a  new 
missile  or  pod,  the  more  likely  the  user  will  want  to  use 
it  on  other,  perhaps  unsuitable,  aircraft.  The  resulting 
vibration  data  must  be  available  to  the  designers  to 
ensure  it  is  suitable  for  the  actual  environment.  These 
stmctural  test  articles  can  also  be  used  to  gain  other 
needed  data  such  as  incremental  drag,  changed 
handling  qualities  and  flutter  characteristics.  The 
resulting  vibration  data  must  be  available  to  the 
designers  to  determine  if  the  missile  or  pod  is  suitable 
for  the  intended  operational  environment. 

In  all  cases,  optimal  selection  of  instmmentation 
requires  considerable  engineering  judgment.  It  is 
relatively  easy  to  select  instmmentation  after  thermal 
or  vibration  problems  arise;  the  difficully  comes  in 
predicting  instrumentation  needs  during  the  aircraft 
design  phase.  As  the  program  progresses  through  the 
development  process,  the  R&M  engineers,  along  with 
thermal  and  vibration  specialists,  should  continuously 
refine  their  instmmentation  requirements. 

4.10  Instrumentation  Data  Processing 

Complex  tools  are  also  needed  to  reduce  special 
instmmentation  data  to  usable  form.  Engineers  from 
many  different  disciplines  normally  share  these  tools. 
This  means  that  R&M  engineers  have  help  on  the 
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planning  process.  But,  the  R&M  engineering  interests 
are  different  than  other  disciplines  and  the  R&M  people 
must  be  fully  involved  in  the  planning  process  to  make 
certain  that  their  unique  requirements  are  addressed.  In 
the  recent  past  large  mainframe  computers  where  used 
almost  exclusively.  The  current  trend,  however,  is 
towards  smaller  machines  such  as  engineering 
workstations  for  at  least  part  of  the  task.  Much  has  been 
written,  in  AGARD  volumes  and  elsewhere,  on  the 
process  of  converting  raw  instrumentation  data  into 
engineering  units.  This  treatise  will  rely  on  that 
previous  work.  Once  the  instrumentation  data  are 
available  in  engineering  units  form,  the  flight  test  R&M 
engineer  must  select  and  present  the  appropriate 
environmental  data  in  a  suitable  form.  The  two  most 
important  aspects  of  environmental  data  are  thermal 
and  vibration  and  will  be  discussed  separately. 

Planning  must  ensure  that  data  reduction  tools  have  the 
capability  to  present  information  in  the  formats  needed 
by  R&M  engineers.  Most  thermal  data  have  the 
advantage  of  requiring  a  low  sample  rate  because  of  the 
slow  rates  of  temperature  change  of  any  object  with 
noticeable  mass  (and  resultant  high  thermal  inertia). 
The  problem  in  dealing  with  thermal  data  from  aircraft 
is  that  a  given  temperature  may  be  a  function  of  marty 
variable  such  as  airspeed,  altitude,  ambient  air 
temperature  and  throttle  setting.  As  with  aircraft 
performance  parameters,  thermal  data  must  be 
corrected  to  "standard  day  conditions"  where  possible. 
If  correction  is  not  possible  (usually  because  of  the  lack 
of  a  thermal  model),  the  data  must  be  presented  with 
all  conditions  and  caveats  noted. 

When  the  aircraft  has  a  specific  design  mission  it  is 
possible  to  show  the  temperatures  throughout  the 
mission.  This  technique  is  useful  if  a  reliability 
problem  is  thought  to  be  temperature  induced.  Another 
useful  tool  is  the  temperature  mapping  technique.  Here, 
the  specific  temperature  is  mapped  throughout  the 
aircraft  envelope.  A  considerable  amount  of  data, 
usually  from  many  flights,  is  required  but  the  results 
are  eloquent  and  portray  much  information  in  a 
compact  form.  Here,  different  characters  are  used  to 
show  different  temperature  ranges.  The  engineer  must 
carefully  select  the  temperature  ranges  to  be  shown. 
For  example,  below  critical  temperatures,  a  single 
character  might  represent  a  wide  range  of  values.  That 
is,  one  symbol  could  be  used  to  indicate  that  the 
measured  temperature  is  in  a  region  of  indifference  or, 
at  least,  in  a  satisfactory  state.  Above  satisfactory 
temperatures,  different  symbols  should  be  selected  to 


show  severity  of  problems.  Normally  this  means  using 
increments  of  10  degrees  Celsius.  If  color  media  is 
available,  the  problem  with  displaying  discrete 
increments  vanishes. 

Both  of  these  presentation  techniques  are  suitable  for 
use  on  an  engineering  work  station  when  the  data  are 
available  in  engineering  units.  Because  of  the  cost,  in 
both  instramentation  and  engineering  man-hours,  these 
techniques  are  generally  limited  to  suspected  problems, 
high  cost  items  and  safety  of  flight  issues. 

Vibration  data  requires  a  very  high  sample  rate  and  is 
correspondingly  more  difficult  to  reduce.  Again, 
engineering  workstations  are  being  used  once  the  raw 
data  are  converted  to  engineering  units.  As  with 
temperature,  vibration  levels  change  with  a  number  of 
variables.  Generally,  it  is  important  to  know  the  peak, 
or  worst  case  vibration  level  and  a  measure  of  the 
average  level.  A  common  data  presentation  scheme  is 
to  plot  peak  acceleration  as  a  function  of  frequency. 
Figure  5  is  an  example.  A  second  technique  is  called  a 
power  spectral  density  (PSD)  plot.  Here,  the 
acceleration  squared  divided  by  the  frequency  is 
plotted  as  the  abscissa  while  the  frequency  is  plotted  as 
the  ordinate.  Because  vibration  is  often  related  to  the 
dynamic  pressure  on  the  aircraft,  a  third  method  plots 
acceleration  versus  dynamic  pressure.  The  example 
shown  in  Figure  6  uses  the  common  Enghsh  units 
(pounds  per  square  foot).  This  particular  method  has 
the  advantage  of  eUminating  several  variables  but 
requires  more  data  reduction  effort. 

As  with  thermal  data,  the  associated  costs  often 
prohibit  analysis  of  vibration  data  except  for  suspected 
problems,  high  cost  components  and  safety  of  flight 
issues. 

4.11  Safety 

Safety  must  be  a  primary  consideration  for  all  flight  test 
activity.  Although  R&M  evaluations  are  not  the  most 
hazardous  tests  conducted,  safety  is  still  very  wortlty 
of  concern.  Quoting  in  part  from  AGARDograph 
AG-300-Vol.  8:  "The  secret  to  accident  prevention  is 
anticipating  personnel  mistakes,  equipment 
malfunctions  and  environmental  aberrations  which 
change  hazards  —  to  accidents."  Like  R&M,  safety  is  a 
designed-in  characteristic.  Failure  to  design-in  safety 
leads  to  the  need  for  procedures  and  equipment  to 
assure  safety.  Climatic  extremes  exaggerate  hazards. 
Again  quoting  from  AG  -300-Vol.  8:  "Working  around 
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as  well  as  operating  a  flight  vehicle  in  extreme  climatic 
conditions  compounds  hazards  and  demands  constant 
attention  to  safety.  Like  flight  itself,  extreme  or  adverse 
environmental  conditions  are  unforgiving  of  the 
ill-prepared,  the  complacent  and  the  uninformed." 

A  relatively  common  R&M  test  that  requires  particular 
attention  to  safety  is  the  quick-turn  demonstration  (also 
called  a  hot  turn).  'Hiis  demonstration,  and  the 
development  leading  to  it,  is  intended  to  measure  the 
minimum  time  between  recovery  from  one  sortie  and 
launchfor  the  next  sortie.  For  fighter  aircraft  this  means 
simultaneous  refueling  and  weapons  reload.  For  these 
type  demonstrations,  safety  considerations  are 
essential.  Before  the  total  demonstration  is  attempted, 
each  subpart  should  be  completed  many  times.  Once 
each  subtask  is  optimized  for  time  and  safety,  then  the 
total  effort  should  be  walked  through  and  carefully 
studied.  In  the  interest  of  safety,  most  of  the  subtasks 
should  be  simulated  during  these  walk  throughs.  For 
example,  a  fuel  tmck  should  be  positioned  as  planned 
for  the  real  quick  turn  The  refuehng  hose  should  be 
correctly  placed  and  connected  and  disconnected  at  the 
appropriate  points  in  the  walk  throughs,  but  fuel  should 
not  be  transferred  until  enough  walk  throughs  have 
been  performed  to  assure  that  safety  is  not 
compromised. 

Quick  turns  and  other  R&M  tests  or  demonstrations 
with  a  potential  safety  impact  should  always  be 
accomplished  in  this  "build-  up"  marmer.  This  means 
that  sufficient  schedule  time  must  be  included  in  the 
test  program  to  allow  these  tests. 

A  useful  practice  before  a  potentially  hazardous  test  is 
a  peer  review.  The  individuals  plaiming  the  test  should 
present  their  plans  to  a  peer  group  not  immediately 
associated  with  the  test  at  hand.  Then  planners  should 
discuss  any  hazards  that  have  been  identified  and  the 
procedures  designed  to  minimize  such  hazards.  The 
reviewers  should  search  for  unforeseen  problems  and 
consider  the  efficacy  of  the  proposed  procedures  to 
minimize  hazards.  Further,  the  planners  should  present 
the  "built-up"  approach  while  the  reviewers  ensure  that 
the  approach  is  logical.  The  results  of  these  reviews 
should  be  fully  documented  and  no  significant 
deviations  allowed  without  further  review. 


4.12  Joint  Reliability  and  Maintainability 
Evaluation  Teams  fJRMET) 

A  final  planning  effort  should  include  establishment  of 
a  group  to  participate  in  classification  of  R&M  data  and 
review  of  results.  The  basic  nature  of  R&M  data  drives 
the  need  for  such  a  group. 

In  marty  respects,  R&M  data  are  more  subjective  than 
data  from  oAer  engineering  disciplines.  Often,  it  is  not 
clear  that  an  anomaly  is  an  inherent  defect  or  was 
somehow  induced.  Similarly,  there  is  often 
disagreement  as  to  the  criticahty  of  failures.  Further, 
contracts  often  contain  definitions  of  failure  that  are 
significantly  different  than  those  normally  used  by  the 
operating  command. 

The  U.S.  Air  Force  A-10  development  contract  defined 
a  failure  as  a  loss  of  mission  critical  function  that 
occurred  after  the  end  of  the  preflight  inspection  and 
before  the  start  of  the  postflight  inspection.  That  veiy 
artificial  definition  excluded  all  noncritical  failures  and 
many  failures  not  discovered  in  flight.  The  user’s 
definition  of  failure  included  all  maintenance  actions 
needed  to  correct  inherent  defects.  Both  definitions 
were  used  to  develop  a  MTBF.  As  a  result,  the  weapons 
system  advocates  reported  a  MTBF  over  ten  times 
higher  than  the  more  independent  testers.  This  large 
discrepancy  surfaced  at  an  important  program  review 
and  the  chairing  flag  rank  officers  were  notably 
displeased. 

Adding  more  confusion  is  the  often  large  difference 
between  the  flight  test  environment  and  the  eventual 
usage  environment.  Considerable  engineering 
judgment  is  needed  to  translate  flight  test  results  to 
expected  fleet  results. 

Establishing  a  team  is  a  way  to  obtain  consensus  and 
increase  understanding  of  the  somewhat  subjective 
results.  If  agreement  cannot  be  reached  on  aU  issues,  at 
least  points  of  disagreement  can  be  isolated  and 
order-of-magnitude  disparities  eliminated. 

Acceptance  of  R&M  results  is  maximized  if  all 
program  participating  agencies  are  represented  on  the 
JRMET  (sometimes  called  a  scoring  conference).  This 
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includes  government  program  management, 
contractors,  test  agencies,  independent  oversight 
agencies,  and  support  agencies  (repair  depots).  If  the 
govermnent  is  providing  a  significant  amount  of 
equipment  to  be  integrated  into  a  vehicle,  the  supplying 
government  agency  should  also  be  represented. 

This  group  should  be  formed  before  test  planning  is 
complete  in  order  that  the  test  can  be  stractured  such 
that  all  participant’s  objectives  can  be  met.  The 
JRMETs  can  be  formally  chartered.  Such  a  charter 
should  list  roles  and  responsibilities.  An  example 
charter  is  included  in  Appendix  B. 

Once  flight  testing  begins,  the  group  should  meet 
periodically  (perhaps  monthly)  to  classify  new  data  and 
review  results.  The  details  of  the  classification  process 
are  discussed  in  the  following  test  conduct  section.  One 
way  to  conduct  the  data  review  is  to  provide  all  group 
members  a  computer  listing  of  new  (since  the  last 
review)  R&M  data  for  review  prior  to  a  formal  meeting. 
The  listing  should  be  provided  enough  in  advance  that 
team  members  can  get  information  needed  for 
classification  decisions.  Particularly,  the  contractors 
must  have  sufficient  time  to  provide  prehminary  failure 
analysis. 

Some  test  programs  are  using  computer  networks  to 
provide  team  members  with  ready  access  to  R&M  data. 
Team  members  use  remote  computer  terminals  to 
review  data  from  their  home  office  prior  to  the  meeting. 
At  meetings,  computer  screen  data  are  projected  in 
large  enough  scale  for  participants  to  conduct 
discussions  and  reach  agreements. 

5.0  TESTCONDTirT 

5,1  Initial  Inspection 

When  the  first  and  subsequent  aircraft  are  delivered  to 
the  test  site,  an  exhaustive  physical  inspection  should 
be  conducted.  The  purpose  of  these  inspections  is 
twofold:  to  inventory  or  baseUne  the  aircraft  and  to 
ensure  the  aircraft  is  truly  flight  worthy. 

The  aircraft  inventory  and  baseline  are  conducted  to 
find  what  equipment  was  delivered  with  the  aircraft.  In 
the  early  stages  of  aircraft  programs,  equipment 
shortages  are  common  and  the  testers  must  understand 
how  the  aircraft  is  configured.  The  inventory  begins 
with  a  careful  review  of  all  associated  paperwork.  The 
individual  aircraft  paperwork  should  list  all  deviations 


to  the  baseline  design  If  the  manufacturer  was 
authorized  to  not  comply  with  any  specific 
requirements,  such  waivers  should  also  be  listed  in  the 
aircraft  paperwork.  These  deviations  and  waivers  must 
be  of  sufficient  detail  to  allow  the  test  agency  to 
conduct  the  inspections  needed  to  ensure  that  flight 
safely  is  not  degraded. 

One  U.S.  Air  Force  aircraft  type  was  delivered  with  the 
''blanket''  waiver  stating:  "Certain  clearances  within 
the  Itydraulics  and  flight  controls  subsystems  are  not  to 
military  standards. "  This  very  general  statement  did  not 
provide  enough  detail  to  allow  the  aircraft  to  be 
properly  inspected  for  potential  control  systems 
binding  or  chafing.  As  a  result,  throughout  the  test 
program  the  aircraft  was  plagued  by  a  host  of  minor 
chafing  problems. 

The  using  command  did  not  get  off  so  easy.  As  least 
one  aircraft  was  lost  in  fleet  service  because  of  control 
system  cable  binding.  To  prevent  further  losses,  special 
inspection  procedures  were  set  up,  certain  aircraft 
compartments  were  painted  white  to  ease  inspection 
and  painfully  rigorous  foreign  object  damage  processes 
were  used.  The  best  solution  would  have  been  to  design 
in  adequate  clearance.  When  that  was  not  possible, 
each  potential  interference  point  should  have  been 
carefully  noted  and  inspections  planned.  The  added 
up-front  effort  would  have  been  cheaper  than  an 
aircraft. 

When  the  paperwork  is  satisfactory,  the  aircraft  should 
undergo  a  careful  review  to  ensure  that  the  hardware 
matches  the  "as-delivered"  configuration  information. 
This  usually  means  removing  all  access/inspection 
panels  and  possibly  large  units  such  as  engines.  The 
part  numbers  and  serial  numbers  of  major  line 
replaceable  units  should  be  recorded  along  with  the 
hour-meter  readings  on  those  units.  The  hour  readings 
will  be  needed  to  calculate  time  to  failure  measures  for 
those  units  that  fail.  The  part  and  serial  numbers  are 
essential  to  understanding  the  aircraft  configuration  as 
modifications  are  made  to  the  units. 

Along  with  inventorying  equipment  and  recording 
data,  the  aircraft  should  be  thoroughly  inspected  for 
design  and  workmanship  errors.  Test  aircraft  are 
normally  produced  on  "soft"  toohng  with  prehminary 
blueprints  by  technicians  that  are  just  beginning  to 
learn  the  processes  needed.  The  probability  of  error  is 
very  high.  This  inspection  should  be  done  by  the  most 
experienced  maintenance  technicians  available.  While 
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marty  defects  can  be  found  easily,  good  inspection 
equipment  such  as  lights,  mirrors  and  borescopes  will 
increase  the  percentage  of  problems  found.  Modem 
fiber  optic  borescopes  with  self-contained  illumination 
and  remotely  controlled  steerable  heads  greatly 
improve  access  to  restricted  areas.  Some  current 
borescopes  provide  a  video  output  for  easy  recording 
of  inspection  results. 

If  video  recording  is  available,  it  is  an  economical 
medium  for  recording  defects  found.  Still  photography 
can  also  be  used.  Figure  7  shows  an  example  of  a  wiring 
defect  discovered  during  an  initial  inspection.  Both 
cases  show  wiring  chafing  against  aircraft  stmcture.  In 
the  airborne  vibration  environment,  insulation 
wear-through  and  eventual  electrical  shorts  were 
inevitable.  Besides  those  two,  37  other  wiring/cabhng 
defects  where  noted  and  corrected  on  that  specific 
aircraft.  Figure  8  is  a  somewhat  different  defect.  An 
engine  bay  fire  warmng  loop  assembly  is  resting  on 
aircraft  stmcture.  When  vibration  and  chafing  wears 
through  the  line  the  engine  fire  warning  will  activate, 
the  pilot  will  declare  an  inflight  emergency  and  use  the 
fire  extinguishing  agent.  After  landing,  the  problem 
would  be  found,  but  the  labor  to  clean  up  the 
extinguishing  agent  will  delay  the  test  program 
noticeably. 

After  the  inspection  and  required  fixes,  it  was  felt  that 
the  aircraft  was  safe  to  fly.  Before  ten  sorties  were 
completed,  the  pilot  declared  an  inflight  emergency 
after  many  flight  control  failure  indications.  When  the 
aircraft  was  dismantled,  the  evidence  told  a  clear  story , 
a  wire  bundle  had  chafed  through  and  started  an  inflight 
fire  in  an  area  that  had  not  been  mspected. 

Evidence  of  defects  such  as  these  must  be  fed  back  to 
the  manufacturer  to  prevent  recurrence.  Aircrews 
usually  have  strong  opinions  on  matters  such  as  inflight 
fires  and  can  be  counted  on  to  lend  their  voices  to  the 
efforts  to  convince  the  manufacturer  to  improve  his 
product. 

As  noted,  even  the  initial  inspection  offers  an 
opportunity  to  develop  improved  maintenance 
practices.  After  the  inflight  fire,  the  initial  inspection 
was  considerably  more  rigorous.  Also,  the  irutial 
inspection  is  the  first  chance  to  note  access  problems. 


s  7  Srhpriiiled  Maintenance/Servicing 

Frequently  performed  tasks  such  as  pre/postflight 
inspections  often  consume  fifty  percent  of  the 
maintenance  labor  hours  on  military  aircraft.  For 
reliable  transport  type  vehicles,  the  figure  is  even 
higher.  During  the  fleet  life  of  these  vehicles,  frequent 
actions  may  be  performed  millions  of  times.  Because 
of  this  high  number,  even  the  slightest  labor  and  time 
savings  can  be  important  over  the  service  life  of  the 
vehicle. 

Early  in  the  test  program  engineers  and  experienced 
maintenance  personnel  should  carefully  scrutinize 
these  commonly  occurring  tasks.  Videotape  recordings 
can  show  where  task  flow  might  be  improved  and  time 
saved.  Desktop  computers  with  process  flow  analysis 
software  can  also  help.  Portions  of  tasks  that  cause 
difficulty  might  be  improved  with  more  traimng  or 
different  tools.  One  recent  analysis  showed  a 
noticeable  improvement  in  task  times  when  the 
maintenance  technicians  were  simply  provided  a  better 
quahty  flashhght  to  perform  aircraft  interior 
inspections. 

Besides  performing  tasks  in  the  optimal  manner,  the 
value  of  performing  the  particular  task  should  be 
questioned.  This  is  particularly  true  of  inspections. 
Each  step  in  any  inspection  should  be  carefully 
reviewed  to  establish  a  firm  need  for  the  task.  Often  the 
tendency  is  to  inspect  if  there  is  any  doubt  as  to  the 
possible  need.  This  over-inspection  adds  to  downtime 
and  labor  costs.  Sometimes,  over-inspection  may 
increase  the  probability  of  maintenance  error  and 
actually  detract  from  the  capability  of  the  vehicle. 

If  an  inspection  or  preventive  maintenance  task  is 
needed,  the  task  frequency  should  be  limited  to  that 
essential  to  safe  and  economical  operation.  Often,  the 
inspection  interval  for  newly-designed  aircraft  is 
relatively  short  because  of  the  many  uncertainties  with 
new  vehicles.  This  is  often  prudent,  but  a  plan  should 
be  developed  to  increase  the  intervals  as  data  are 
gathered  and  uncertainties  are  eliminated.  For  example, 
it  is  good  practice  to  perform  Spectroscopic  Oil 
Analysis  on  the  fluids  in  new  aircraft  even  if  it  is  not 
specifically  recommended  by  the  manufacturer.  This 
should  be  done  before  the  first  flight  and  after  the  first 
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several  flights.  The  fluid  metallic  content  measurement 
provides  a  potential  early  warning  of  problems.  But, 
once  the  metallic  content  is  seen  to  be  satisfactory,  a 
maximum  interval  should  be  developed. 

Once  a  scheduled  maintenance  task  is  considered 
optimally  stmctured,  a  standardized  time  should  be 
agreed  to  and  predictions  updated.  The  standardized 
time  can  then  be  used  in  simulations  and  models. 

Scheduled  maintenance  tasks  must  also  be  considered 
in  climatic  extreme  conditions.  Underthoseconditions, 
times  will  likely  increase  significantly.  Sometimes  it 
will  be  necessary  to  change  procedures  to  accomplish 
maintenance  tasks.  Those  special  procedures  should 
also  be  optimized  and  technical  data  changed 
appropriately.  Resulting  times  should  also  be 
standardized  and  used  in  appropriate  models  and 
simulations.  In  this  marmer,  aircraft  maintenance 
requirements  in  extreme  conditions  can  be  predicted. 
AGARDograph  Volume  8,  Series  No.  300  also 
discusses  flight  testing  under  extreme  climatic 
conditions. 

5.3  Unscheduled  Maintenance/Reliability 

Unscheduled  maintenance  is  that  work  required  to  fix 
failures  (real  or  suspected).  That  work  is  the  source  of 
flight  test  reliability  data  and  measures.  During  the  test 
program  thousands  of  failures  may  occur.  Some  will  be 
more  significant  than  others.  If  the  program  was 
properly  planned  and  well  implemented,  the  R&M 
engineers  should  not  need  to  spend  much  time  on  the 
day-to-day  data  collection  efforts.  Instead,  the  test 
engineers  should  focus  on  classifying  and  analyzing 
results.  With  thousands  of  failures  to  consider,  the 
engineers  must  have  some  order-of-battle  to  stmcture 
their  efforts.  The  following  discussion  is  not  the  only 
method  of  classifying  R&M  data,  but  has  proven 
helpful  on  a  number  of  programs. 

An  ideal  reliabihty  evaluation  would  analyze  and  fix 
every  failure  mode  that  occurs.  Practical  limitations 
like  time  and  moneys  always  make  that  impossible. 
First,  fix  the  problems  that  affect  safety  of  flight.  Then, 
those  items  that  prevent  mission  completion  must  be 
corrected.  Lastly,  those  noncritical  failures  that  affect 
availabihty  and  cost  should  be  addressed. 

This  order  of  precedence  should  guide  engineers 
during  the  test  program.  For  every  failure  that  occurs, 
decide  if  there  is  any  scenario  where  the  failure  would 


decrease  safety.  This  is  not  as  obvious  as  it  looks.  If  the 
failure  occurred  during  clear  daytime  flight,  what 
would  have  been  the  impact  if  the  problem  arose  during 
night  or  adverse  weather  conditions?  What  if  this  had 
been  a  second  failure?  For  weapons  delivery  platforms, 
what  if  the  failure  had  occurred  during  the  stress  of 
combat  conditions?  Aircrew  experience  is  essential 
during  these  deliberations.  The  R&M  test  engineers 
must  regularly  consult  with  operations  personnel  to 
understand  the  significance  of  failures. 

Do  not  neglect  ground  operations.  Toxic  materials, 
high  pressure  systems,  and  ordinance  are  a  few  of  the 
hazards  that  have  caused  lethal  accidents  in  the  past. 
Every  failure  must  be  reviewed  to  consider  the 
potential  safety  imphcations  on  ground  operations. 

If  a  failure  does  not  affect  safety,  it  may  still  be  mission 
critical.  For  every  failure,  the  astute  R&M  engineer 
must  decide  if  the  failure  would  prevent  any  of  the 
aircraft’s  designed  missions.  Experienced  aircrew 
must  also  be  consulted  in  these  determinations. 
Further,  many  flight  occurrences  lead  an  aircrew  to 
believe  a  failure  has  occurred,  but  later  investigation 
carmot  find  any  failure.  As  much  as  30  percent  of 
aircrew  reported  avionic  problems  cannot  be  traced  to 
a  failed  component.  But,  if  the  aircrew  believes  a 
failure  has  occurred,  then  the  crew  wUl  act  on  that 
belief.  That  may  include  a  mission  deviation  or,  more 
seriously,  a  mission  abort.  If  the  crew  does  decide  to 
continue  the  mission,  they  will  likely  quit  using  the 
suspect  equipment  and  resort  to  a  less  capable 
alternative  equipment.  This  means  that  a  mission 
critical  anomaly  has  occurred  although  no  hardware 
failure  exists.  This  situation  must  be  treated  in  much 
the  same  way  as  a  hard  failure;  i.e.  the  problem  must 
be  investigated  and  eliminated. 

The  least  significant  failure  category  is  that  class  of 
defects  that  have  no  safety  affect  and  do  not  prevent 
mission  accomplishment,  but  still  consume  time  and 
resources  to  correct.  These  problems  can  be  considered 
in  two  broad,  general  classes:  deferrable  and 
nondeferrable.  The  nondeferrable  problems  are  those 
that  must  be  corrected  before  the  aircraft  can  be 
dispatched  on  the  next  mission.  Deferrable  problems 
can  be  postponed  until  a  scheduled  maintenance 
period. 

Obviously,  those  failures  that  prevent  an  aircraft  launch 
must  be  fixed  first.  Deferrable  problems  must  be 
considered  in  light  of  the  total  amount  of  time  that 
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aircraft  is  unavailable  for  use.  For  example,  simple 
maintenance  tasks  such  as  resealing  panels  become 
nontrivial  when  the  sealant  cure  time  is  long  (as  much 
as  24  hours  sometimes). 

In  summary,  failures  should  be  considered  in  four 
general  classes:  safety-critical,  mission-critical, 
noncritical-nondefenable  and  noncritical-deferrable. 
While  safety-critical  failure  modes  must  be  eliminated 
(and  usually  are),  the  less  severe  failure  modes  should 
be  fixed  if  cost  effective.  Generally,  not  enough 
problems  are  fixed.  That  is,  more  front-end  investment 
in  reliability  improvement  would  lower  life  cycle  costs. 
History  does  not  document  a  single  instance  of  a 
militaiy/aerospace  product  with  excess  reliability 
(measured  in  life  cycle  cost). 

During  program  conduct  each  failure  should  be 
classified  as  to  severity  as  discussed  before.  Also,  each 
failure  should  be  classified  as  to  cause:  i.e.,  inherent 
defect,  induced  defect  or  no  defect.  The  latter  class 
includes  those  cases  where  no  actual  failure  is  ever 
found.  The  computerized  maintenance  database  should 
be  capable  of  storing  and  recalling  these  classifications. 
The  database  should  also  be  capable  of  recalling  all 
data  associated  with  these  categories. 

For  databases  with  those  abilities,  a  most  useful 
computer  product  is  a  hsting  of  the  most  common 
failing  items  in  each  category.  With  such  a  hsting,  it  is 
easy  to  see  where  the  engineering  emphasis  must  be 
placed  to  improve  the  vehicle.  The  test  R&M  engineer 
should  review  this  "high-bumer"  listing  to  insure  that 
the  most  frequently  failing  items  are  being  corrected. 
Similarly,  lists  of  the  items  receiving  the  most  induced 
failures  and  the  most  no-defect  type  failures  must  be 
reviewed  to  insure  that  all  are  being  corrected. 

As  the  test  program  progresses,  the  R&M  engineer 
must  search  for  the  cause  of  failures  and  advocate  any 
action  needed  to  prevent  recurrence  of  those  failures. 
Once  a  particular  part  is  deemed  worthy  of  further 
investigation,  the  action  to  be  taken  depends  on  the 
tools  available  to  the  engineer.  If  flight  test 
instrumentation  is  available  to  measure  the 
environmental  stresses  (like  temperature  and 
vibration),  a  quick  course  of  action  may  be  to  review 
existing  data  to  insure  that  the  equipment  is  not  being 
damaged  by  the  environment.  If  the  environment 
proves  benign  or  instrumentation  is  not  available,  a 
detailed  failure  analysis  (much  like  an  autopsy)  may 
provide  the  cause  of  failure.  If  the  failure  analysis 


points  to  an  enviromnental  overstress  it  may  be 
necessary  to  add  instrumentation  to  measure  the 
operating  environment.  This  is  a  lengthy  process,  but 
needed  if  the  problem  is  severe  enough.  If  the  failure 
cause  is  overstress,  the  needed  fixes  are  also  difficult. 
Either  the  failed  part  must  be  made  more 
environmentally  durable  or  the  operating  environment 
must  be  improved. 

For  heat  related  problems,  sometimes  cooling  air 
allocations  can  be  changed  and  more  or  cooler  air 
suppUed  to  the  problem  component.  But,  this  lowers 
the  coohng  available  to  the  rest  of  the  system  unless  air 
conditioning  subsystem  modifications  are  made.  If 
modifications  are  made  to  increase  cooling  capacity, 
then  more  energy  (such  as  bleed  air  or  electrical  power) 
will  be  needed. 

For  vibration  related  problems,  improved  isolation  or 
energy  absorption  devices  may  attenuate  the  failure 
inducing  energy  enough  to  improve  reliability. 
Unfortunately,  such  devices  consume  space  and  add 
weight. 

Forturrately,  most  (as  much  as  90  percent)  failures  can 
be  prevented  from  recurring  by  eliminating  internal 
design  and  manufacturing  errors. 

For  those  problems  that  are  not  considered  inherent 
defects  but,  induced  failures,  the  objective  is  the  same: 
prevent  recurrence.  The  most  common  causes  of 
induced  failures  are  aircraft  operation  outside  design 
limits  (pilot  error),  mainterrance  induced  damage 
(maintenance  error)  and  secondary  failures.  When 
errors  occur,  the  normal  reaction  is  to  find  the  guilty 
and  inflict  sufficient  punishment  to  insure  more  care  in 
the  future. 

But,  this  approach  only  insures  that  the  currently  guilty 
do  not  repeat  the  offense.  Prevention  of  future  errors 
requires  a  proactive  action.  Anytime  a  pilot  or 
maintainer  commits  an  error,  the  flight  test  engineers 
must  search  for  the  underlying  reason.  Potential 
reasons  for  human  error  include  unfriendly  hardware 
design,  trap-ridden  software,  etc.  Many  errors  have 
more  than  one  factor.  Review  of  aircraft  accident 
reports  shows  the  spectmm  of  things  that  can  lead  to 
human  error  and  shoirld  be  mandatory  reading  for  the 
novice  flight  tester. 

An  oft-advanced  fix  for  human  error  is  to  improve  the 
training  given.  Often  this  is  the  only  cost  effective  way 
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to  eliminate  problems  once  an  aircraft  is  in  fleet  usage. 
During  flight  test  it  may  be  possible  to  change  the 
design  to  remove  the  source  of  potential  error  and  this 
option  should  always  be  considered  first. 

In  some  cases,  tasks  are  so  difficult  that  the  probability 
of  error  is  significant  even  if  utmost  caution  is  used.  In 
some  early  versions  of  the  F-16  aircraft  there  was  a 
requirement  to  tighten  a  steel  oil  line  to  a  steel  fitting 
at  a  torque  of  over  100  foot-pounds.  That  torque  figure 
was  appropriate  for  the  steel  fittings  used.  The  steel 
fitting  however,  was  threaded  into  a  magnesium 
gearcase.  The  technical  instructions  carefully  specified 
how  wrenches  were  to  be  used.  But  access  to  and 
visibility  of  the  area  was  almost  nonexistent.  As  a 
result,  when  tightening  the  steel  line,  the  maintainers 
would  accidentally  transfer  the  100  foot-pounds  of 
torque  into  the  magnesium  gearcase.  On  the  ground, 
the  stripped  threads  were  not  apparent.  In  flight  the 
engine  vibration  quickly  loosened  the  stripped  fitting, 
the  oil  drained  oveiboard,  and  the  aircraft  generator 
froze.  Usually  the  emergency  power  unit  functioned 
properly  and  the  pilot  landed  the  vehicle.  After  ten  such 
instances,  the  design  was  changed  to  prevent  such 
accidental  overtorquing.  Here,  the  safety  of  flight  issue 
made  an  error-proof  design  essential  and  the  hardware 
was  modified. 

Not  all  induced  failures  are  the  result  of  human  error. 
Secondary  failures  are  those  failures  that  occur  when 
associated  equipment  fails  and  induces  other  failures. 
Sometimes,  the  original,  or  primary  failure,  may  be 
insignificant  compared  to  the  resulting  secondary 
failure.  Consider  the  case  where  a  small 
forward-mounted  fastener  loosens  and  departs  the 
aircraft-through  the  engine.  Here  the  primary  problem 
(the  lost  fastener)  was  induced  by  human  error 
(insufficient  torque)  and  the  secondary  failure  was 
catastrophic. 

In  another  memorable  case,  a  thyratron  (high  power 
switching  device)  failed  and  killed  eight  other 
electronic  components.  The  tl^ratron  was  inexpensive 
(U.S.  $200)  and  had  a  low  expected  reliability  of  about 
1,000  hours  mean  life.  One  of  the  secondary  failures 
was  an  expensive  pulse  forming  i^twork  that  should 
have  been  ailure-free.  The  design  was  changed  to 
prevent  secondary  failures. 

In  summary,  all  induced  failures,  whether  caused  by 
human  error  or  system  design,  must  be  investigated  for 


cause  and  a  reasoned  decision  made  to  alleviate  the 
problem. 

The  last  class  of  problem  is  considered  the  no-defect 
failure.  This  seemingly  paradoxical  situation  arises 
when  the  aircrew  or  the  built-in-test  reports  a  problem 
and  maintenance  cannot  cause  the  problem  to  repeat  or 
find  any  failed  components.  The  situation  where  the 
aircrew  reports  a  problem  and  maintenance  finds  no 
defect,  but  the  symptoms  repeat  until  a  problem  is 
eventually  found  is  not  within  this  no-defect  class. 
Instead,  this  is  a  maintainability  problem.  This 
no-defect  anomaly  arises  more  often  than  normally 
expected.  Experience  shows  that  about  30  percent  of 
all  reported  avionic  faults  result  in  no  failed  component 
being  found.  On  some  aircraft,  the  no-defect  rate  is 
considerably  higher. 

In  the  expert’ s  opinion,  most  intermittent  avionic  faults 
are  caused  by  defective  connections  in  electrical 
cormectors.  This  is  a  particularly  insidious  failure  mode 
to  find.  When  the  electrical  connector  is  demated  for 
troubleshooting  or  inspection,  the  problem  normally 
vanishes.  In  fact,  any  movement  of  the  electrical 
cabling  or  cormector  may  be  sufficient  to  improve  the 
cormection  enough  to  restore  the  system  to  operation. 
This  problem  exists  with  all  electronic  systems,  but  is 
particularly  acute  with  aircraft  because  size  and  weight 
considerations  lead  to  the  smallest  possible  connectors. 
Gold-plated  contact  surfaces  are  used  to  maximize 
conductivity  and  minimize  oxidation,  but  connectors 
remain  a  problem. 

The  best  way  to  minimize  connector  problems  is  to 
minimize  cormectors  (not  surprising).  The  F-4  series 
aircraft  had  905  electrical  connectors  while  the  more 
recently  designed  F-18  (by  most  measures,  the  most 
rehable  U.S.  fighter)  has  808  cormectors.  The 
significant  decrease  in  cormector  count,  combined  with 
the  increased  avionic  capability  of  the  F-18,  shows  that 
careful  design  can  help.  The  flight  test  community  must 
always  search  for  ways  to  simplify  aircraft— cormectors 
included.  Flight  test  must  also  carefully  scrutinize 
cabling,  cormectors,  and  supports  for  mechanical 
integrity.  Vibration  prone  cabling/cormectors  are 
failure  prone  connectors. 

5.4  Unscheduled  Maintenance/Maintainabilitv 

The  maintainability  analysis  methods  used  for 
scheduled  maintenance  also  work  for  unscheduled 
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maintenance.  An  important  difference  between  the  two 
categories  of  maintenance  is  troubleshooting.  The 
more  complex  the  system,  the  harder  the  fault  isolation 
process.  Electronic  systems  with  more  than  one  LRU 
"blackbox"  often  require  more  time  to  fault  isolate  than 
to  physically  repair.  Even  the  most  experienced 
technicians  carmot  isolate  the  failed  LRU  in  some  older 
aircraft  systems.  When  that  occurs,  all  suspected  LRUs 
must  be  replaced  (sometimes  called  shotgun 
maintenance).  The  cost  of  this  inefficiency  make  it 
imperative  that  R&M  engineers  fix  such  problems. 
Many  aircraft  contracts  require  a  fault  isolation 
capability  that  will  isolate  any  failure  to  a  single  LRU. 
The  isolation  capability  may  consist  of  troubleshooting 
instructions  (technical  data),  ground  test  equipment, 
built-in-test,  or  any  combination.  During  the  test 
program,  any  failure  that  carmot  be  isolated  must  be 
studied  in  detail.  The  R&M  engineers  must  report  the 
failure  mode,  fault  isolation  problems  and  recommend 
the  changes  needed  to  improve  the  isolation  process. 

5.5  Contractual  Requirements  Verification 

An  important  (and  very  visible)  element  of  the  flight 
test  is  verifying  that  the  contractor(s)  achieved  the 
R&M  levels  specified  in  contracts.  The  process  is  made 
more  difficult  by  the  wide  variety  of  R&M  measures 
used  in  different  contracts.  Further,  every  contract 
seems  to  reinvent  circumstances  which  must  be 
excluded  from  consideration.  Generally,  the  purpose  is 
to  assign  liability  for  all  occurrences  caused  by  the 
contractor’s  design  and  manufacturing  efforts. 
Conversely,  hability  is  withheld  for  occurrences 
beyond  the  contractor’s  control.  The  concept  is  simple 
in  theory  and  difficult  in  practice. 

For  example,  maintenance  technician  error  is  beyond 
the  control  of  the  contractor  and  excluded  from 
consideration.  However,  if  the  error  was  induced 
because  the  technician  followed  incorrect  instractions 
in  contractor-provided  technical  data,  the  occurrence 
must  be  considered.  An  even  more  subtle  problem 
occurs  when  the  contract  specifies  a  joumeymen-level 
technician  must  be  able  to  perform  all  maintenance 
tasks.  Then,  when  maintenance  error  occurs,  there  is  an 
inevitable  dispute  over  whether  the  task  is  to  difficult 
for  a  journeymen  technician.  Contracts  seldom  discuss 
such  subtle  nuances  which  must  be  negotiated  with  the 
contractor.  The  JRMET  or  equivalent  group  is  an 
excellent  fomm  for  such  negotiations.  The  results  must 
be  carefully  documented  throughout  the  test  program. 


During  the  test  program  every  occurrence  must  be 
assessed  for  chargability  against  reliability  and 
maintainability  guarantees.  That  is,  all  reported  defects 
and  maintenance  actions  must  be  reviewed  to 
determine  if  they  are  attributable  to  the  contractor’s 
design  or  manufacturing  processes.  Again,  the  JRMET 
or  it’s  analog  is  the  correct  forum  for  such 
classification.  Once  a  decision  has  been  made,  each 
action  should  be  tagged  as  to  the  contractual  relevance. 
This  tagging  or  classification  should  be  entered  into  the 
computer  database. 

5.6  Built-in-Test 

Built-in-test  (BIT)  is  a  widespread  technique  used  to 
help  in  fault  isolation.  This  self  diagnosis  capability  is 
also  used  to  aid  aircrews  by  showing  that  critical 
equipment  is  working.  Aircraft  contracts  often  state 
something  like;  "Built-in-test  shall  detect  100  percent 
of  all  mission  critical  failures  and  isolate  90  percent  of 
all  detected  failures  to  a  single  LRU."  False  alarms 
(incorrect  indication  of  failure)  are  always  a  problem 
and  contracts  normally  speciiy  a  maximum  false  alarm 
rate.  Final  development  and  assessment  of  the  BIT 
capability  is  an  important  part  of  the  R&M  test  effort. 

All  reported  failure  indications  and  maintenance 
actions  must  be  assessed  as  to  the  success,  failure  or 
nonapplicabiUty  of  BIT  in  each  instance.  Once  again, 
this  is  a  JRMET  task.  The  results  of  the  assessment 
should  be  entered  into  the  computer  database.  When 
the  database  is  complete,  calculation  of  BIT  measures 
of  merit  is  straightforward. 

5.7  Summary 

Much  of  the  practicing  R&M  engineer’s  time  is  spend 
classifying  reported  failures  and  maintenance 
occurrences.  Reported  failures  must  be  classified  as  to 
:  safety  criticality,  mission  criticality,  cause,  and  BIT 
effectiveness.  Also,  contractor  attributabihly  must  be 
determined.  Once  the  classification  process  is  complete 
and  the  computer  database  is  updated,  calculating 
R&M  measures  of  merit  is  easy. 

6.n  DATA  ANALYSTS  AND  PRESENTATION 

Besides  data  classification,  data  analysis  is  an  ongoing 
activity  throughout  the  test  program.  The  previous 
chapter  briefly  discussed  the  "high-bumer"  lists  of 
most  frequently  failing  parts  and  emphasized  the  need 
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to  find  and  fix  problems.  This  improvement  process  is 
the  real  "value-added"  of  a  good  RAM  test  program 
and  all  data  analysis  should  support  that  objective. 

6.1  Production  Readiness 

After  finding  and  fixing  problems,  determination  of 
production  readiness  is  the  highest  payback  effort.  An 
important  program  milestone  is  the  beginning  of 
production.  It  is  certainly  the  most  pohtically  visible. 
Production  startup  means  committing  large  amounts  of 
money  to  the  program  and  is  almost  an  irreversible 
commitment  to  the  system.  The  flight  test  results  are 
often  scrutinized  and  criticized  in  the  technical  and 
political  arenas.  The  political  champions  of  the 
weapons  system  will  loudly  broadcast  positive  flight 
test  results  and  dowiplay  any  negative  results.  The 
weapons  system  opponents  will  seize  any  shortcoming 
discovered  during  test  and  herald  it  as  evidence  of 
developer  incompetence.  If  there  is  no  negative  news, 
the  opponents  will  likely  question  the  test  data  and  in 
extreme  cases,  question  the  integrity  of  the  testers. 

The  production  readiness  question  is  particularly 
difficult  for  the  Reliability  and  Maintainability 
engineers  because  of  the  reliability  growth  phenomena. 
If  the  R&M  engineers  have  been  successful,  the  system 
reliability  has  steadily  improved  during  the  test 
program.  More  over,  marty  problems  that  were 
discovered  during  test  could  likely  not  be  economically 
fixed  on  the  test  vehicles,  but  could  readily  be  fixed  in 
the  production  versions  of  the  aircraft  The  challenge 
is  to  predict  the  reliability  and  maintainability  of  the 
production  aircraft  using  data  from  test  versions  of  the 
vehicle.  This  must  be  done  rigorously  to  fend  off  the 
inevitable  criticism  from  those  disappointed  in  the 
results. 

One  method  is  shown  graphically  in  Figure  9.  That 
method  of  plotting  reliabihty  growth  on  log-log  scales 
is  called  the  Duane  reliability  growth  model  (Reference 
1).  This  illustration  first  plots  the  reliability  growth 
dtrring  the  F-20  test  program.  The  first  portion  of  the 
plot,  between  100  and  645  cumulative  flight  hours, 
shows  the  actual  reliability  measured  during  flight  test. 
Figure  9  shows  the  measured  cumulative  reliability  as 
1.9  hours  meantime  between  maintenance(inherent)  at 
the  645  flight  hour  points.  Also  shown  is  the  measured 
reliability  growth  rate  (alpha)  of  0.25.  The  area  of  the 
plot  between  645  and  10,000  flight  hours  is  aprojection 
based  upon  the  measured  0.25-growth  rate.  The 


0.25-growth  rate  is  conservative  and  generally 
considered  low-  risk. 

More  controversial  is  the  step  improvement  shown  at 
the  645  flight  hour  point.  That  step  represents  the 
improvement  expected  from  programmed  changes. 
Table  1 1  lists  the  changes  planned.  The  cross-hatched 
area  in  Figure  9  represents  the  uncertainty  in  estimating 
the  reliability  improvement.  This  type  of  presentation 
is  a  method  to  project  future  reliability  from  current  test 
results.  The  obvious  weakness’  of  estimating  the 
reliability  of  improvements  and  assuming  a  growth  rate 
can  and  are  criticized.  The  advantage  is  that  these 
assumptions  are  visible  and  can  be  discussed  and 
justified  in  detail.  Commonly  the  improvements  are 
obviously  low  risk.  For  example.  Table  1 1  lists  radial 
tires  as  an  improvement.  The  reliability  improvement 
of  radial  over  bias  ply  tires  is  well  established  and  that 
particular  change  should  not  be  questioned.  Other 
changes,  such  as  the  digital  displays,  are  harder  to 
estimate  and  justify  a  reliability  improvement.  If  the 
improvement  is  a  completely  new  development,  the 
initial  reliability  may  be  lower  than  the  proceeder  item. 
Then  it  is  necessary  to  plan  a  reliability  growth  for  the 
"improved"  item. 

Table  11 

F-20  Plaimed  Reliability  Improvements 

•  Digital  Displays 

•  40  KVA  Generator 

•  Main  Battery 

•  Fuel  Quantity  Indicator 

•  Radial  Tires 

•  Display  Processor 

•  Inertial  Navigation  Unit 

•  Tail  Beacon 

•  Windshield  Sealant _ 

Similarly,  maintainability  must  be  predicted  from 
flight  test  data.  Where  major  changes,  such  as  shown 
in  Table  12,  are  plarmed,  a  growth  plot  and  step 
improvement  is  appropriate. 

The  preceding  assumes  that  the  system  is  ready  for 
production  or  at  least  close.  The  situation  is  much  more 
difficult  if  it  is  not  ready  for  productioa  The  onus  is 
usually  on  the  testers  to  clearly  prove  that  the  system 
is  not  suitable  for  production  and  what  changes  are 
needed.  Recent  U.S.  programs  such  as  low-altitude 
navigation  and  targeting  infrared  for  night  (L  ANTIRN) 
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and  the  advanced  medium-range  air-to-air  missile 
(AMRAAM)  have  suffered  production  delays  until 
reliability  has  been  improved  and  demonstrated.  In 
these  cases,  the  political  arena  became  interested  and 
the  onus  was  on  the  developer  to  prove  reliability  was 
sufficient. 

Table  12 

F-20  Planned  MaintainabiUty  Improvements 

•  Onboard  Oxygen  Generation  System 

•  Engine 

ignitor  box 
support  equipment 

•  Airframe 

hinge  radome 
bleed  air  duct 
radar  air  dehydrator 

6.2  Specification  Verification 

Demonstrating  manufacturer  comphance  with 
contracts  and  specifications  is  one  of  the  more 
contentious  aspects  of  flight  test  R&M  evaluations. 
The  manufacturer  is  strongly  motivated  to  demonstrate 
that  the  aircraft  meets  all  R&M  requirements.  If  the 
testers  claim  otherwise,  they  had  best  have  complete 
evideiKe  to  show  any  noncompliance.  The  first  area 
questioned  will  likely  be  data  collection  accuracy. 
Once  the  testers  prove  that  the  data  are  accurate,  the 
data  interpretation  will  be  scmtinized.  This  means  that 
the  data  analysis  must  be  rigorous  to  withstand 
challenge.  Often,  specifications  are  both  qualitative 
and  quantitative  for  R&M. 

Qualitative  reliability  is  usually  easy  to  verify  or 
disprove.  A  single  occurrence  of  a  proscribed  action 
will  show  noncompliance.  Some  qualitative  reliability 
requirements  can  be  verified  by  inspection  of  the 
design  and  aircraft.  These  requirements  often  take  the 
form  of  specifying  certain  functional  characteristics 
such  as:  "All  electrical  connectors  shall  provide 
protection  from  any  anticipated  environment." 
Inspection  will  show  if  environmental  coimectors  are 
installed.  The  problem  becomes  more  difficult  if  an 
environmental  cormector  fails  to  provide  adequate 
protection.  A  photograph  would  demonstrate  that  the 
cormector  did  not  provide  protection,  but  some 
evidence  of  the  environment  at  time  of  failure  is  also 
needed.  Obviously,  a  cormector  might  be  designed  to 


withstand  a  certain  humidity  level,  but  would  not 
function  long  if  submerged. 

Other  qualitative  reliability  requirements  are  not  as 
readily  demonstrable.  A  common  flight  controls 
system  requirement  is:  "No  single-point  failure  shall 
cause  the  probability  of  aircraft  loss  to  exceed  one  part 
in  a  million."  A  careful  review  of  the  design  will  help 
verify  that  the  requirement  has  been  met,  but  it  is 
possible  to  overlook  subtle  details.  As  flight  test 
proceeds,  the  subtleties  should  become  evident.  In  one 
classic  case,  a  quad-redundant  flight  control  system 
had  four  power  rectifiers  using  a  common  ground  wire. 
Fortunately,  the  situation  was  discovered  before  the 
wire  failed. 

Qualitative  maintainability  requirements  are  also 
amenable  to  inspection.  These  requirements  are 
designed  to  ease  maintenance  and  to  prevent 
maintenance  error.  Some  common  requirements  are: 

1.  Captive  fasteners  shall  be  used  for 
nonstmctural  access  covers. 

2.  Keyed  electrical  connectors  shall  preclude 
mating  to  wrong  receptacles. 

3.  No  adjustment,  other  than  operator  controls, 
shall  be  required  when  replacing  one  or  more  units. 

There  are  many  more  common  qualitative 
requirements;  the  testers  should  develop  a  checklist  to 
ensure  all  requirements  are  considered  as  the  program 
progresses. 

Considerable  care  is  required  to  verify  quantitative 
reliability  requirements.  The  simplest  and  most 
common  measure  of  reliability  is  MTBF.  Both  time  and 
failure  must  be  carefully  and  precisely  defined.  As 
discussed  previously,  time  (or  measurement  of  stress) 
is  commonly  defined  as  flight  time.  The  flight  test 
environment  complicates  the  issue.  For  many  reasons, 
flight  test  aircraft  are  likely  to  be  operated  on  the 
ground  more  than  a  fleet  service  aircraft  would  be.  This 
"excess  ground  operating  time"  adds  stress  to  mar^ 
aircraft  components.  If  the  aircraft  ground  cooling 
capability  is  not  as  good  as  the  inflight  cooling,  it  is 
possible  that  the  ground  environment  is  more  stressful 
then  flight.  Provision  must  be  made  to  account  for  the 
"excess  ground  operating  time. "  The  simplest  way  is  to 
simply  censor,  or  exclude  failures  that  are  obviously 
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caused  by  ground  operatioa  Another  method  is  to 
consider  a  ground  operating  hour  as  equivalent(  in 
stress)  to  some  fraction  of  a  flight  hour.  Then,  an 
"equivalent  flight  hour"  can  be  calculated  and  used  as 
a  total  measure  of  stress: 

Equivalent  Fhght  Hours  =  Flight  Hours  +  "Excess 
Ground  Hours"  /  Adjustment  Factor 

The  enviroiunental  stress  factors  on  U.S. 
Mil-Handbook  217  are  commonly  used  for  the 
adjustment  factor  in  the  previous  equation.  The 
handbook  lists  6.0  as  the  ratio  of  stress  difference 
between  the  airborne  uninhabited  environment  and  the 
ground  fixed  environment.  This  translates  as  saying 
that  flight  is  6.0  times  as  stressfiil  as  ground  operation. 
Once  equivalent  flight  hours  are  calculated,  that  figure 
can  be  used  as  the  numerator  in  the  various  "Meantime 
Between"  equations. 

Failures  must  also  be  precisely  defined.  The  earlier  data 
classification  section  discussed  the  different  types  of 
failures  and  failure  criticality.  Specifications  must 
clearly  state  what  is  considered  a  failure  for  aity 
requirement. 

A  last  factor  causing  difficulty  in  assessing  quantitative 
reliabihty  is  the  reliability  growth  phenomena.  It  is  well 
established  that  reliability  will  improve  throughout  the 
development  program  if  an  effort  is  made  to  correct  and 
prevent  recurrence  of  failure  modes.  If  reliability  is 
improving  as  a  function  of  test  time,  at  what  point 
should  reliability  be  measured?  The  reliability  shown 
at  the  end  of  test  will  be  the  most  representative  of  that 
experienced  infleet  service.  But,  should  the  cumulative 
reliabihty  to  date  be  used  or  perhaps  a  point  estimate  at 
the  end  of  the  test  program?  Should  improvements 
planned  but  not  implemented  be  included  to  estimate 
reliabihty? 

Answers  to  these  questions  as  well  as  definitions  of 
stress  time  and  failure  must  be  fully  addressed  before 
attempting  to  assess  specification  compliance.  Ideally, 
the  fhght  test  agency  and  the  contractor  would  agree  to 
all  definitions  before  the  start  of  test. 

7.n  REPORTING 

7.1  General 

The  most  exhaustive  test  would  have  no  value  if  the 
results  are  not  used.  The  first  step  to  usage  is  to  make 


the  results  known.  This  chapter  discusses  some  ways 
to  report  test  results. 

7.2  Deficiencies 

All  deficiencies  must  be  reported  when  possible.  The 
long  lead  times  needed  to  implement  changes  make  it 
cmcial  that  defects  be  reported  in  a  timely  manner  and 
corrective  actions  begun  as  soon  as  possible.  Fhght 
safety  is  also  a  major  reason  to  spread  the  word  about 
problems.  If  a  deficiency  has  any  safety-of-flight 
impact,  the  information  must  be  disseminated 
immediately.  Aviation  history  lists  many  cases  where 
aircraft  accidents  would  have  been  prevented  if  the 
right  people  had  the  needed  information. 

Many  organizations  have  developed  admimstrative 
systems  to  manage  these  technical  problems.  These 
systems  consist  of  a  form  to  capture  all  needed 
information  and,  perhaps,  a  computerized  database  to 
store  the  information.  If  possible,  a  computerized 
system  is  most  highly  recommended.  Lengthy  test 
programs  on  complex  aircraft  produce  hundreds  and 
sometimes  thousands  of  deficiency  reports. 

Figure  10  is  an  example  form  completed  to  show  the 
details  of  an  aircraft  rehability  problem.  Table  13  lists 
and  explains  the  data  elements  on  the  form.  As  shown, 
the  deficiency  report  focus  is  on  a  single  narrow 
problem.  Another  important  point  to  note  is  the  level 
of  detail.  The  information  must  be  sufficient  to 
convince  readers  that  a  problem  tmly  exists  and  that  the 
problem  impact  is  sufficient  to  warrant  a  corrective 
action.  Often  the  aircraft  advocates  and  contractors  are 
not  pleased  when  told  of  problems  with  their  prize 
creation  and  the  flight  test  engineer  must  present  a 
convincing  argument.  Analogous  to  a  barrister  arguing 
a  case,  the  engineer  must  present  compelling  evidence 
to  prove  the  existence  and  impact  of  deficiencies. 

Any  computerized  data  system  should  also  track 
actions  after  the  deficiency  is  initially  reported.  Some 
important  information  includes  details  of  corrections 
and  implementation  effectively.  Also,  any  additional 
flight  test  efforts  needed  to  either  better  define  the 
problem  or  to  show  the  effectiveness  of  fixes  should  be 
included. 

These  deficiency  reports  should  be  considered  action 
documents.  This  means  the  time  between  discovery  of 
the  problem  and  the  report  should  be  a  matter  of  days 
(hours  if  flight  safety  is  affected).  Every  report  should 


SERVICE  REPORT  RECOR 


SUBJECT 

□  Category  I  Service  Report 
Q  Category  II  Service  Report 

□  Repeat  Category  I/ll  Service  Report 

□  Initial  Acceptance  Inspection 


□  Yes  Ei!  No  Source  Selection  Sensitive 

(Protect  I.A.W.  APR  70-15) 
S  Mission  Essential  Deficiency 

□  Degrades  Mission  Deficiency 

□  Proposed  Enhancement 


Excessive  Failure  Rate  of  Armament  Control  Panel  (A-6  Card) 


ASC/ENR 


13.  OPERATING  TIME  AT  FAILURE 

Various 


14.  GOVT.  FURNISHED  PROPERTY 
□  YES  □  NO 


15.  QUANTITY 
RECEIVED 


16.  DEFICIENT  ITEM  WORKS  ON  OR  W! 

END  ITEM  I 


17.  DOLLAR  VALUE 


18.  ESTIMATED  CORRECTION  COST 


19.  ITEM  UNDER  WARRANTY 
□  YES  □  NO 


— 

1.  FROM 

452  FLTS/EN 

3.  REPORT  CONTROL  NO. 

F-15CTF-94-001 

4.  DATE  DISCOVERED 

18  Jan  1993 

5.  NSN/FSN 

□  UNK 

5870-00-341-4221 

□  N/A 

6.  NOMENCLATURE 

Armament  GOntrol  Panel 

7.  MFR.  SHIPPER,  OVERHAUL 

□  UNK 

Polvmornhic  Svstems 

□  N/A 

8.  MANUFACTURER'S  PART  NO. 

□  UNK 

C-935869-09 

□  N/A 

9.  SERIAL,  LOT,  OR  BATCH  NO. 

□  UNK 

Various 

□  N/A 

10.  CONTRACT,  PURCHASE  ORD,  DOC  NO. 

□  UNK 

Various 

□  N/A 

11.  ITEM 

□  UNK 

NEW  □  OVERHAULED 

□  N/A 

12.  DATE  MFD,  REPAIRED  OR  OVERHAULED 

□  UNK 

Various 

□  N/A 

NEXT  HIGHER  ASSEMBLY 


21.  ACTION  OR  DISPOSITION 

X]  HOLDING  EXHIBIT  . 


□  RELEASED  TO 


20.  WUC  OR  PSEUDO  WUC 

75QED 


SEDTO  □  RETURNED  TO 


22.  DETAILS  (CONTINUE  ON  REVERSE) 

The  Armament  Control  Panel  has  failed  six  times  during  the  flight  test  program  (140 
hours,  MTBF=  27  hours).  In  all  cases  the  Master  Jettison  circuult  card  assembly  (A-6) 
had  failed.  Replacement  of  that  card  restored  the  failed  control  panels  tp 
operation.  Physical  examination  of  the  failed  A-6  cards  revealed  no  obvious  cause. 
The  conformal  coating  was  intact  and  in  good  condition. 


A.  SUBJECT  AREA 


B.  IMPACTS  ON 


C.  HAZARD  CODE 
(MIL-STD-882A) 

□  I  □  II 

□  III  □  IV 


0.  RECOMMENDATIONS  (CONTINUE  ON  REVERSE) 

A  detailed  physics-of4failure  analysis  required  to  determine  the  exact 

f ailurecause .  The  last  two  failed  A-6  cards  are  being  held  by  the  test  team. 
Recommend  that  these  cards  be  returned  to  the  manfacturer  (  or  an  independent  test 
laboratory)  for  detailed  analysis. 


E.  STANDARD  REPORT  DESIGNATOR 


G.  ORIGINATOR  NAME  &  PHONE  NO 


F.  COMMAND  CODE 


H.  RELEASER  AUTHORITY  NAME,  PHONE 


AFTO  form  240 

JAN  79 


PREVIOUS  EDITION  IS  OBSOLETE 


Figure  10  AFTO  Form  240 
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Table  13 

Deficiency  Report  Contents 


Block 

Block 

Block 

No. 

Title 

Contents 

none 

Subject 

Identifies  type _ of  report 
and  lists  the  mission 
criticality. 

none 

Title 

One  sentence  narrative 
describing  the  deficiency. 

1 . 

Fro.m 

Name  and  address  of  agency 
reporting  the  deficiency. 

2  . 

To 

Name  and  address  of  agency 
responsible  for  correcting 
the  deficiency. 

3  . 

Report 

Control 

No. 

Number  assigned  for  admin¬ 
istrative  tracking  and 
control  purposes. 

4  . 

Date  Dis¬ 
covered 

Calendar  date  the  defi¬ 
ciency  was  found. 

5  . 

NSN/FSN 

Part  nu-mber  unique  to  US 
military. 

6  . 

Nomen¬ 

clature 

Name  of  the  deficient  part. 

7  . 

MFR, 

SHIPPER 

OVERHAUL, 

Agency  responsible  for 
manufacturer,  repair  or 
transDortation  (if  damaged 
in  transit  )  of  deficient 
part . 

8  . 

MANUFACT¬ 

URER 

PART  NO. 

Unique  number  assigned  by 
the  manufacturer. 

9  . 

SERIAJL.DOT 
BATCH  NO. 

Number  to  help  identify 
deficient  part. 

10. 

CONTPJiCT, 

PURCHASE 

ORDER 

Number  assigned  to  defi¬ 
cient  part  purchase  paper¬ 
work  . 

11 . 

ITEM 

Indicates  if  deficient  part 
was  new  or  overhauled. 

12  . 

DATE  MFD. 
REPAIRED 

OR  OVER¬ 
HAULED 

Shows  date  deficient  item 
was  last  renewed. 

13  . 

OPERATING 
TIME  AT 
FAILURE. 

Measure  of  accumulated 
stress  at  failure. 

14  . 

GOVT  FUR¬ 
NISHED 
PROPERTY 

Shows  if  deficient 
item  was  government 
provided . 

Block 

No. 

Block 

Title 

Block 

Contents 

15  . 

QUANTITY 

Lists  number  of  similiar 
parts  available  and  numb( 
deficient . 

16  . 

DEFICIENT 
ITEM  WORKS 
ON/WITH 

Lists  type  aircraft  and 
next  level  of  assembly 
above  deficient  part. 

17  . 

DOLLAR 

VALUE 

Cost  of  deficient  part. 

18  . 

ESTIMATED 

CORRECTION 

COST 

Self-explanatory. 

19  . 

ITEM  UlTOER 

Used  to  process  warranty 
claims . 

20  . 

WUC  OR 
PSEUDO 

WUC 

Hardware  identifer  code 
used  in  addition  to  part 
numbers . 

21  . 

ACTION  OR 
DISPOSI¬ 
TION 

Shows  what  was  done  with 
deficient  part. 

22  . 

DETAILS 

Most  important  entry  on 
form.  A  narrative  of  pr 
lem  and  impact .  Must  hav 
enough  information  to  al 
correction  to  be  designe 

A. 

SUBJECT 

AREA 

Lists  area  of  deficiency 
(Design,  Logistics,  etc. 

B. 

IMPACTS  ON 

Lists  technical  area  i.m- 
pacted.  by  deficiency  (RS, 
Utility,  etc. ) . 

C. 

HAZARD 

CODE 

Shows  safety  impact  of 
deficiency. 

D. 

E&F. 

RECOMMEN¬ 

DATIONS 

Narrative  decribing  any 
potential  remedy  for  the 
deficiency. 

Military  designators  un; 
to  US  Air  Force. 

G&H 

Identitv  of  person  repo; 

ing  deficiency. 
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cause  a  conscious  decision  made  to  either  fix  the 
problem  in  some  matmer  or  to  live  with  it.  If  the 
problem  cannot  be  fixed,  the  information  should  not  be 
discarded.  Instead,  all  such  information  should  be 
carefully  archived.  Every  aircraft  is  modified  during  its 
service  life  and  it  may  be  possible  to  fix  the  problems 
thea 

7.3  Progress  Reports 

Several  progress  or  interim  reports  are  usually  needed 
during  the  test  program.  These  are  often  coincident 
with  different  program  milestones  such  as  production 
decisions.  Customer  needs  should  establish  the 
contents  of  these  reports.  To  the  extent  possible,  these 
reports  should  be  considered  during  the  program 
planning  phase.  Report  times  and  content  should  be 
plarmed. 

Specifically,  for  production  decisions,  the  program 
advocates  will  need  to  show  how  well  the  aircraft  is 
doing  in  relation  to  contractual  and  user  requirements. 
These  program  advocacy  milestones  may  take  the  form 
of  briefings  to  decision  makers  or  summary  level  issue 
papers.  Ineither  case,  limited  time  and  space  mean  only 
the  important  issues  can  be  discussed.  For  example,  for 
production  decision  briefings,  R&M  is  seldom 
allocated  more  than  two  briefing  slides.  As  discussed 
earlier,  reliability  and  maintainabihty  growth  curves 
will  usually  be  needed  to  show  that  the  vehicle  is 
making  satisfactory  progress  towards  requirements. 

Also  recommended  is  a  list  of  deficiencies  discovered 
and  the  plarmed  corrections.  This  list  can  be  rank 
ordered  with  the  most  serious  first.  Such  a  list  can 
convince  various  parties  that  the  problems  are  being 
fixed.  Often,  the  problems  are  reported  in  the  news 
media,  but  the  fixes  seldom  are.  This  type  of  candor 
establishes  the  credibiUty  of  the  testers  while 
correcting  popular  misconceptions. 

7.4  Final  Technical  Reports 

Traditionally,  a  final  report  is  written  at  the  conclusion 
of  the  program.  These  reports  are  not  normally  action 
documents.  By  the  end  of  the  test  program,  production 
is  well  rmderway  and  the  time  to  make  improvements 
is  past.  The  final  report  should,  however,  sununarize 
the  test  program  including  problems  encountered  and 
corrected.  The  report  should  also  present  a  complete 
list  of  R&M  parameters  measured  and  current  at  the 


end  of  the  flight  test  program.  This  includes  growth 
projections  as  discussed  earlier.  Table  14  is  an  outline 
of  one-way  a  final  R&M  report  might  be  structured. 

7.5  Lessons  Learned 

Some  organizations  have  a  formal  repository  of 
so-called  lessons  learned.  These  "lessons"  are  intended 
to  document  problems  so  that  others  may  avoid 
unpleasant  relearning  experiences.  Suitable  lessons 
learned  subjects  may  come  from  any  phase  of  the  flight 
test  effort:  plaiming,  provisioning,  execution  or 
analysis/reporting.  Also,  any  portion  of  the  aircraft  or 
associated  equipment  may  provide  a  lesson  that 
designers  should  learn  from. 

An  excellent  example  of  a  programmatic  lesson  learned 
and  ever  relearned  is  the  need  for  good  failure  analysis. 
History  does  not  offer  one  instance  of  overanalyzing 
failures.  To  the  contrary,  too  many  failures  are  written 
off  as  "random"  and  needed  improvements  left  undone. 

7.6  Technical  Society  Papers 

Many  technical  societies  exist  for  every  engineering 
disciphne,  R&M  included.  Often  these  societies 
publish  journals  and  hold  technical  symposia.  These 
are  excellent  opportunities  for  the  individual  engineer 
(or  small  team)  to  get  publication  credit  and  spread  the 
word  on  important  technical  work.  Many  society 
publications  are  saturated  with  papers  from  academia 
and  practical  papers  are  often  welcomed. 

8.0  FOLLOW-UP 

When  the  reports  are  written,  the  aircraft  delivered  to 
the  customer  and  the  test  team  scattered  to  the  four 
winds,  there  is  still  more  to  do.  The  effectiveness  of  the 
R&M  evaluation  can  only  be  measured  by  examining 
the  user’s  success  in  operating  the  system.  The  more 
problems  the  user  has,  the  less  effective  the  flight  test 
program  was. 

8.1  Accident  Reports 

Test  agencies  should  routinely  examine  accident 
reports  concerning  systems  they  tested  and  others. 
When  an  accident  occurs,  the  testers  must  ask 
themselves  if  they  could  have  done  anything  that  would 
have  prevented  that  particular  occurrence.  Even  if  they 
where  not  the  responsible  test  agency,  there  are  lessons 
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Table  14 

Reliability  and  Maintainability  Report  Outline 


PREFACE;  Relation  of  this  report  to  other  reports  and  work  in  progress 

EXECUTIVE  SUMMARY:  A  summary  of  the  report  with  a  brief  description  of  the 
results,  conclusions  and  recommendations 

table  of  CONTENTS: 

list  of  ILLUSTRATIONS;  Including  growth  charts,  task  timelines  and  photo¬ 
graphs/drawings  to  illustrate  problems 

list  of  TABLES:  Including  R&M  measures  of  merit 

INTRODUCTION: 


Background:  Historical  information  such  as  preceding  tests,  program 
authorizations  and  direction 

General:  Test  dates,  hours  flown,  planned  versus  actual  tests  and  criti¬ 
cal  issues  and  questions 

Test  Objectives:  Focus  of  tests 

Test  Limitations:  Any  limitations  which  prevented  com.plete  accomplishment 
of  test  objectives 

Test  Item  Description;  Brief  description  of  the  article  tested--  Differ¬ 
ences  between  the  test  article  and  planned  production  vehicle  should  be  care¬ 
fully  listed--  If  a  description  is  overly  long  and  might  distract  from  the 
focus  of  the  report,  an  appendix  might  be  used  to  describe  the  article 

TEST  and  EVALUATION: 

Test  Methods:  Including  brief  data  collection,  processing  and  analysis 
overview  --  refer  to  appendix  for  detailed  description 
Test  Results; 

Overall  Aircraft  Results: 

Individual  siibsystem  Results: 

Both  aircraft  level  and  subsystem  results  should  include  RSH  measures,  de¬ 
scriptions  of  deficiencies  (refer  to  previous  deficiency  reports) ,  and  needed 
illustrations  such  as  growth  plots,  task  timelines  and  photographs/drawings  to 
show  problems,  include  confidence  levels  as  appropriate,  draw  conclusions  and 
make  recommendations  as  appropriate  and  discuss  needed  changes  and  any  further 
testing  required. 

CONCLUSIONS  and  RECOMMENDATIONS :  A  summary  of  important  conclusions  and 
recommendations  in  the  body  of  the  report 

REFERENCES :  Include  sources  for  specifications,  requirements,  designs,  and 
other  helpful  reports  or  docu.me.nts 

APPENDICES :  Include (as  needed)  detailed  descriptions  of  test  methods,  test 
articles  and  results/data.  Sortie  or  Missions  Summaries,  Deficiency  Reports 
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to  be  learned  from  every  accident.  The  reports  contain 
a  wealth  of  detail  that  can  help  improve  and  justify 
future  test  efforts. 

8.2  In-Service  R&M  Data 

Test  Agencies  should  also  examine  the  in-service 
R&M  results  from  the  aircraft  that  they  and  others 
tested.  Comparisons  of  actual  usage  with  test  program 
results  will  quickly  point  out  any  weakness  with  the  test 
process  and  allow  better  tests  in  the  future.  This  is 
particularly  important  when  test  results  are  used  to 
extrapolate  to  predict  mature  system  R&M 
performance.  As  discussed  earlier,  the  test  program 
should  significantly  improve  the  R&M  of  the  aircraft. 
If  further  improvement  is  expected  in  the  production 
version  of  the  aircraft,  such  expected  improvement 
should  be  verified  by  analysis  of  in-service  data. 

Sometimes  test  agencies  must  review  in-  service  R&M 
data  as  a  protective  measure.  If  the  fleet  R&M  results 
are  significantly  less  than  measured  during  the  test 
program,  the  tester’s  credibihty  is  questioned.  During 
the  F-15A  acquisition  program  the  aircraft  central 
computer  was  specified  to  have  a  2,000-hour  MTBF 
and  measured  1,846-hour  MTBF  during  flight  test. 
After  the  production  aircraft  were  delivered,  the 
in-service  results  showed  a  44-hour  MTBF  for  the 
computer.  The  drastic  difference  and  the  high  computer 
cost  quickly  got  the  attention  of  many  flag  rank  officers 
and  the  testers  were  questioned.  When  the  flight  test 
results  were  easily  defended,  the  question  shifted  to  ask 
why  in-service  results  where  so  bad.  After  a  detailed 
analysis,  the  answer  was  apparent.  The  software  was 
still  evolving  and  the  computers  were  often  removed 
for  reprogramming.  The  in-service  R&M  data 
collection  system  considered  any  removal,  regardless 
of  cause,  as  a  failure  unless  it  was  specifically  told  that 
no  failure  existed.  All  removals  were  traced,  by 
computer  serial  number,  through  all  levels  of  repair  and 
only  those  computers  requiring  physical  repair  were 
classed  as  failures.  After  that  exhaustive  reconciliation 
effort,  the  MTBF  was  recalculated  as  1,780  hours  for 
in-service  use.  The  flag  rank  officers  were  mollified. 
The  testers  gloated  and  gleefuUy  pointed  to  the 
in-service  data  collection  system  as  yet  another 
example  of  user  incompetence. 

8.3  Modification  Requirements. 

Another  measure  of  the  development  program  success 
is  the  need  for  modifications  after  the  aircraft  has  been 


delivered  to  the  user.  The  testers  should  monitor  the 
modifications  needed  to  see  if  problems  went 
urmoticed  during  the  test  program.  Academician  G.  V. 
Novozhilov,  writing  in  the  U.S.S.R.  publication 
VESTNIK,  discussed  reliability  of  wide-body  aircraft. 
As  part  of  his  work,  he  compared  the  complexify  and 
reliabiUty  of  the  11-62  and  11-86  flight  control  systems. 
Quoting:  "One  can  judge  the  complexity  of  the  control 
system  of  a  modem  aircraft  by  the  number  of  units  that 
make  it  up.  Thus,  for  the  11-62  aircraft  control  system 
there  are  some  16  units,  78  for  the  11-86."  Academician 
Novozhilov  continues  to  discuss  the  technique  used  to 
develop  reliable  flight  controls  for  the  11-86.  He 
summarizes  by  comparing  the  modifications  needed  in 
the  first  4  years  of  11-62  service  with  the  first  4  years  of 
11-86  service.  Figure  11  reproduces  that  comparison 
and  clearly  shows  the  improvement. 

9.0  FUTURE  rONSTDERATIONS 

The  easing  of  global  tensions  will  mean  a  continuing 
decline  in  funding  for  military  purposes.  This  alone 
would  translate  to  an  irrereasing  demand  for  reliable 
and  maintainable  equipment.  Coupled  with  the 
emerging  processes  and  technologies,  the  time  is  right 
for  a  drastic  increase  in  R&M  performance. 

9.1  Processes 

In  the  past,  too  much  emphasis  has  been  placed  on 
measuring  R&M  and  not  enough  on  designing  reliable 
and  maintainable  equipment.  This  realization,  along 
with  improved  computer  design  tools,  is  leading  to  a 
shift  in  emphasis.  A  fundamental  way  to  improve 
reliability  is  to  design  equipment  with  sufficient 
integrity  to  thrive  in  the  intended  environment.  New 
thermal  and  vibration  computer  analysis  tools  are 
increasing  the  ability  of  the  designer  to  test  the  design 
before  metal  is  cut.  Anthropometric  computer 
simulations  are  being  used  to  test  the  man-  machine 
interface.  This  will  eliminate  marty  maintainability 
difficulties  before  ary  fabrication  begins. 

9.2  Technologies 

Reliable,  accurate  navigation  systems  have  been  a 
major  aerospace  problem  for  decades.  With  the 
deployment  of  the  global  positioning  system  and  the 
development  of  ring  laser  gyros,  the  solution  is  near. 
Systems  with  5,000-hour  MTBF  are  feasible.  The  price 
and  size  of  these  systems  will  allow  dual  redundancy 
for  essentially  infinite  mission  reliability. 
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Airborne  radars  have  also  been  an  ongoing  problem. 
From  the  days  of  vacuum  tube  technology,  airborne 
radars  have  seldom  exceeded  a  100-hour  MTBF.  A 
major  contributor  to  that  unreliability  was  and  is  the 
transmitter-receiver  (TR).  With  the  emergence  of 
economical  TR  modules  in  phased  array  radars,  the 
mission  reliability  will  be  greatly  improved.  When  a 
single  module  fails,  sensitivity  will  be  reduced,  but  the 
function  will  not  be  lost.  This  "graceful  degradation" 
feature  will  do  much  to  improve  airborne  radar 
reliabihty. 

Additionally,  new  display  technology  will  eliminate 
the  use  of  cathode  ray  tubes  (CRTs)  in  aircraft.  While 
CRT  reliability  has  been  continuously  improved  over 
the  years,  the  very  high  voltages  needed  have  usually 
posed  rehability  problems.  And  CRTs  are  bulky  even 
without  the  packaging  needed  to  ensure  their  survival. 

Radar  data  processors  have  been  the  reliability 
downfall  of  many  systems.  The  ever  increasing 
coiiiputer  power  available  from  single  chip  processors 
will  simplify  the  design  challenge  and  perhaps  some  of 
the  performance  increase  can  be  dedicated  to  redundant 
processing. 

New  avionic  mechanical  packaging  techniques  are 
evolving  to  simplify  on-aircraft  repair  and  improve 
avionic  structural  integrity.  Avionic  cormectors 
continue  to  be  a  vexing  problem.  With  the  ever 
increasing  density  of  pin  connections,  an  electrical 
answer  may  not  be  possible.  Perhaps  increased  use  of 
multiplexed  fiber  optics  will  prove  to  be  an  answer. 

Aircraft  engine  reliability  has  increased  drastically 
over  the  past  two  decades.  Noticeable  improvement 
resulted  from  electronic  fuel  controls  replacing  the 
electromechanical  nightmares  of  days  past.  Advances 
in  avionics  will  mean  further  improvement  in  fuel 
control  function  reliability.  Materials  sciences 
advances  have  also  contributed  to  rotating  machinery 
durabihty  and  more  such  advances  are  approaching 
maturity.  Ceramics  and  metal  matrix  composites  may 


prove  to  be  a  low  cost  solution  to  many  high 
temperature  problems. 

Efforts  are  ongoing  to  achieve  a  more  electric  aircraft. 
This  effort  strives  to  minimize  or  reduce  the  use  of 
airborne  hydraulic,  pneumatic,  mechanical  and 
accessory  gearbox  systems.  These  would  be  replaced 
by  high  reliability  electrical  controllers  and  high 
efficiency  electrical  motors.  Weight  and  space  savings 
are  expected  to  be  accompanied  by  improved  reliability 
and  maintainability. 

Improving  reliability  will  mean  major  changes  in  the 
way  systems  are  operated  and  maintained.  Many 
avionic  units  are  reliable  enough  for  two-level 
maintenance  (on  aircraft  and  depot)  now.  In  the  future, 
most  shop  type  maintenance  will  become  redundant 
and,  some  equipments  will  require  only  single  level 
maintenance  -  they  will  be  discarded  upon  failure. 

The  increased  reliability  has  alrea^  made  it  possible 
to  consider  composite  U.S.  Air  Force  Wings.  For 
decades,  most  wings  were  a  single  aircraft  type  to  take 
advantage  of  the  economies  of  scale  in  the  shop  repair 
environment.  This  greatly  increased  the  command  and 
control  problems  when  different  aircraft  types  were 
needed  for  a  mission.  Now,  it  is  increasingly  feasible 
to  assign  multiple  aircraft  types  to  the  same  wing  and 
greatly  increase  unit  cohesion. 

And,  the  maintenance  process  will  also  change. 
Built-in-test  is  a  continuously  improving  discipline. 
Artificial  intelhgence  tools  like  expert  systems,  fuzzy 
logic,  and  neural  networks  all  hold  promise  to  make 
fault  diagnosis  better  and  cheaper. 

Microelectronics  makes  it  possible  for  each  individual 
unit  to  remember  it’s  own  operations  (including  stress) 
use,  failure/repair  history  and  modification  record. 
Maintenance  malpractice  will  be  easier  to  isolate  and 
correct.  Intermittently  failing  units  will  be  found 
quicker. 
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APPENDIX  A 

RELIABILITY  AND  MAINTAINABILITY  DATA  COLLECTION  ELEMENTS 


This  appendix  discusses  the  different  elements  of  data 
needed  from  maintenance  performed  on  the  equipment 
being  tested.  Figure  A1  is  the  standard  form  used  by 
the  U.S.  Air  Force  Systems  Command.  This  form  is 
completed  for  all  work,  scheduled  and  unscheduled, 
performed  on  the  test  aircraft.  The  form  is  similar  to 
those  used  in  regular  fleet  service,  but  asks  for  the  more 
detailed  data  needed  in  a  test  environment.  Not  all  of 
the  information  on  Figure  A1  is  essential  to  an  R&M 
test  program.  Where  such  entries  are  unique  to  the  U.S. 
Air  Force,  they  are  not  discussed  in  this  appendix. 
Figure  A2  is  a  redrawn  and  translated  version  of  the 
form  used  by  the  U.S.S.R.  for  data  eollection  during 
fleet  service.  The  similarities  are  interesting  to  note. 

CONTROL  NUMBER  (Block  A  on  Figure  AD  -  The 
control  number  is  a  means  of  linking  all  related 
maintenance  actions  into  a  single  event  All  actions 
caused  by  a  single  failrrre  or  flight  discrepancy  have  a 
single  control  number.  With  this  number,  all  related 
actions,  on  the  aircraft  or  at  a  shop,  depot  or  vendor  can 
be  related.  For  much  of  the  U.S.  military,  this  nine  digit 
is  called  ajob  control  number.  The  first  two  digits  (e.g. 
92)  are  the  decade  and  year.  The  next  three  digits  (e.g. 
185)  are  the  Julian  date  that  the  problem  was 
discovered.  The  final  four  digits  are  assigned 
sequentially  beginning  with  000 1  for  the  first)  ob  of  the 
day.  Once  originated,  this  number  is  assigned  to  all 
subsequent  related  work,  regardless  of  when  the  work 
is  done.  When  prefixed  with  a  unique  base  or  location 
code,  the  job  control  number  uniquely  identifies  a 
single  job  within  the  U.S.  military. 

VEHICLE  TYPE  (Block  4  on  Figure  Al)  -  The  type  of 
aircraft  or  munition  (e.g.  F-15E,  GBU-15)  being 
maintained.  Called  the  Model-Design-Series  by  the 
U.S.  Military,  this  element  simply  names  the  vehicle. 

SERIAL  NUMBER  (Block  5  on  Figure  Al)  -  The 
serial  number  of  the  item  being  maintained.  For 
aircraft,  the  tail  number  is  often  used. 

WORK  CENTER  (Block  3  on  Figure  Al)  -  The 
organization  responsible  for  the  maintenance  being 
recorded.  Shows  whether  the  work  was  on-aircraft, 
shop  or  depot.  Can  also  be  used  to  show  which  vendor 
is  repairing  equipment. 


TIME/CYCLES/MILES  (Block  6  on  Figure  Al)  -  The 
measure  of  equipment  usage  or  accumulated  stress. 
Flight  hours  for  aircraft  for  example.  Normally 
rounded  to  the  nearest  whole  number. 

WHEN  DISCOVERED  DATE/TIME  (Block  7  on 
Figure  Al)  -  The  date  and  time  that  the  failure  was 
discovered.  Not  used  for  scheduled  maintenance  or 
servicing  type  actions.  Used  as  a  cross  check  on  the 
Julian  date  component  of  the  control  number  and  to 
calculate  times  between  failures. 

REPORT  DATE  (Block  8  on  Figured!)  -  Date  the 
maintenance  recorded  was  actually  done  (as  opposed 
to  the  date  failure  occurred).  Used  to  calculate 
maintenance  expenditures  per  calendar  period  and 
elapsed  time  out  of  service  statistics. 

EQUIPMENT  CLASS  (Block  9  onFigure  Al  -  part  of 
work  order  number)  -  A  code  showing  if  item  being 
repaired  is  aircraft,  munition,  support  equipment,  etc. 

TYPE  MAINTENANCE  tBlock  9  on  Figure  Al  -  part 
of  work  order  number)  -  A  code  showing  if 
maintenance  was  servicing,  inspection,  repair,  etc.). 
Used  to  calculate  statistics  for  different  types  of 
maintenance. 

WHEN  DISCOVERED  CODE  (Block  11  on  Figure 
AH-  A  code  showing  when  the  failure  was  found  (not 
used  for  scheduled  maintenance)  such  as  preflight, 
in-flight,  etc.)  Used  to  classify  severity  of  failure. 

ENOTNE  POSITION  NUMBER  (Block  12  on  Figure 
Al}  -  A  number  showing  which  individual  is  being 
maintained  when  the  equipment  has  more  than  one  like 
item  installed.  Used  to  identify  failure  trends  as  a 
function  of  equipment  location. 

ACTIVITY  IDENTIFICATION  (Block  13  on  Figure 
Al)  -  A  code  showing  the  geographic  location  where 
maintenance  is  being  performed.  When  combined 
with  the  control  number,  a  mihtary  wide  unique 
number  is  formed. 

MANUFACTURER  (Block  U  on  Figure  Al)  -  The 
manufacturer  of  the  equipment  being  repaired.  Used 
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A.  JOB  CONTROL  B.  PR|  C.  TIME  SPEC  REQD  D.  WORK  AREA  E.  ESTIMATED  MANHOURS 
NUMBER 
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to  identify  problems  unique  to  different  equipment 
makers. 

NOUN  rBlock  1 5  on  Figure  AD  -  The  name  of  the  part 
being  removed  or  repaired.  The  name  is  taken  from  a 
manual  with  standard  names. 

SF.RTAL  NUMBER  (Block  16  on  Figure  All  -  The 
serial  number  of  the  unit  being  repaired  or  replaced. 
This  number  is  essential  to  finding  "bad  actor"  units. 

TTMF7CYCLF.S/MTLES  (Block  17  on  Figure  Al)  - 
The  measure  of  stress  on  the  unit  being  repaired  or 
replaced.  This  accumulated  stress  measure  is  taken 
from  self-contained  elapsed  time  counters. 

PARTNTJMRF.R  tBlock  18  onFigure  Al)  -  A  number 
assigned  by  the  manufacturer.  This  number  is  the  only 
way  to  tell  the  version  of  a  part.  During  development 
programs,  parts  are  often  improved  many  times  and  the 
part  number  changes  with  each  equipment  change.  For 
example,  the  initial  version  might  be  numbered 
191238.  Then,  the  first  change  would  be  191238-1. 
Other  number  schemes  are  used,  but  mostly,  the  higher 
the  "dash  number"  the  later  the  versioa  The  number 
of  the  part  being  repaired  is  essential  for  analyzing 
failure  modes. 

WORK  UNIT  CODE  tBlock  19  on  Figure  Al)  -  The 
Work  Unit  Code  is  a  "hardware  address"  used  by  the 
U.S.  Navy  and  Air  Force  to  hierarchically  define  the 
the  equipment.  For  example,  the  radio  navigation 
system  might  be  coded  as  71000.  The  system  could 
contain  several  functional  capabilities  such  as  TACAN 
and  ILS.  Then,  the  TACAN  would  be  coded  71A00 
and  the  ILS  71B00.  Continuing  the  example,  the 
TACAN  receiver/transmitter  line  replaceable 
unit(LRU)  would  be  coded  71AA0,  the  TACAN 
control  panel  71  ABO  and  the  anteruia  71AC0. 
Finishing  the  example,  the  modules  within  the 
receiver/transmitter  (shop  replaceable  imits  -  SRUs) 
would  be  coded  71  AAA,  71AAB  and  so  forth. 

This  identification  scheme  is  necessary  because  part 
numbers  change  too  frequently  and  are  to  complex  to 
be  useful.  U.S.  Military  Specification  38769C,  A 
Manual,  Technical,  Work  Unit  Code  contains  more 
details. 

SYMPTOM/MALFUNCTION  CODE  fBlock  20  on 
Figure  AD  -  The  code  is  a  numeric  code  to  indicate  the 
symptom  of  the  equipment  initially  and,  when  the 


failure  is  understood,  to  indicate  the  failure  mode 
among  a  class  of  possible  modes.  This  code  is  used  for 
computer  processing. 

Several  classes  of  codes  are  used.  The  first  class 
consists  of  codes  that  describe  inherent  failures. 
Inherent  failures  are  those  failures  caused  by  an 
inherent  defect  in  the  failed  part. 

The  second  class  lists  induced  failure  modes.  That  is, 
those  failures  where  the  defect  was  not  inherent  in  the 
the  failed  part.  Examples  include  secondary  failures, 
maintenance  error  and  weather  damage. 

A  third  class  includes  the  "no-defect"  failure.  This 
occurs  when  a  observed  performance  anomaly  cannot 
be  traced  to  a  broken  part  and  the  symptom  disappears. 

A  final  list  of  codes  is  termed  "product  improvement". 
These  codes  are  used  when  the  maintenance  work  is 
not  being  done  to  repair  a  failure,  but  rather  to  improve 
the  equipment.  Modifications  are  the  common 
example. 

BUn.T-IN-TEST  CODE  IBlock  24  on  Figure  Al)  - 
This  code  shows  the  use  and  accuracy  of  the  aircraft 
built-in-test  for  the  maintenance  being  recorded.  The 
code  contains  two  characters  to  show  how  faults  where 
identified  and  isolated.  Codes  used  on  the  C-17  test 
program  are: 

1.  BIT  detected  a  malfirnction  when  one 
existed. 

2.  BIT  failed  to  detect  a  malfunction  when  one 
existed. 

3.  BIT  correctly  isolated  to  the  failed  line 
replaceable  unit  (LRU). 

4.  BIT  did  not  isolate  to  the  failed  LRU. 

a  BIT  assisted  but  technical  data  and 
support  equipment  where  needed. 

b.  Technical  data  and/or  support  equipment 
where  only  method. 

c.  Procedures  did  not  work. 

5.  BIT  indicated  a  malfunction  where  none 
existed.  (False  Alarm) 
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INSTALLED  ITEM  INFORMATION  blocks  25  to 
29onFiguieAII  -  Installed  item  information  is  needed 
when  an  LRU  is  changed.  The  data  needed  are 
identical  in  nature  to  that  necessary  for  the  equipment 
being  removed  or  repaired. 

DESCRIPTION  OF  DISCREPANCY  (Block  30  on 
Figure  AD  -  The  fault  description  is  one  of  the  more 
critical  data  elements.  Other  codes  and  elements  can  be 
corrected  if  the  narrative  is  accurate  and  complete.  For 
inflight  discovered  faults,  the  aircrew  must  be  carefully 
questioned  to  get  all  of  the  details  recorded.  The  details 
are  important  to  ease  the  maintenance  task  of  finding 
the  problem  and  important  to  the  reliability  engineer  to 
understand  the  problem  and  the  impact. 

SPECIALTY  CODE  IBlock  titled  AFSC  on  Figure 
Al)  -  This  code  describes  the  specialty  of  the 
maintenance  technician  performing  the  work.  Engine 
repair,  avionics  technician  and  sheet  metal  specialist 
are  examples  of  the  different  types  of  maintenance 
personnel.  This  information  is  used  to  calculate  the 
number  of  the  different  types  of  maintenance 
specialists  needed  for  a  mature  aircraft. 

START/STOP  TIMES  IBlocks  32/33  and  35/36  on 
Figure  AD  -  These  are  the  clock  times  when 
maintenance  work  was  initiated  and  terminated.  These 
times,  together  with  the  maintenance  crew  size,  yield 
the  man-hours  needed  for  a  specific  repair  task. 

CREW  SIZE  fBlock  titled  NR  on  Figure  Al)  -  The 
number  of  maintenance  persoimel  working. 

DELAY  CODE  (Blocks  34  and  .35  on  Figure  Al)  - 
Although  not  an  essential  data  element,  the  reason  for 
maintenance  delays  can  be  helpful  in  analyzing 
maintenance  tasks. 

UNITS  IB  lock  40  on  Figure  All  -  A  code  to  show 
whether  the  work  is  complete  after  this  action  is 
finished.  This  is  necessary  because  some  repair  tasks 
continue  over  several  days  and  more  rarely,  several 
weeks. 

ACTION  TAKEN  CODE  IBlock  41  on  Figure  AD  - 
This  entry  is  a  code  showing  the  type  of  repair 
accomplished.  Typical  codes  represent  actions  such  as 


remove  and  replace,clean,  and  troubleshoot.  A  typical 
maintenance  action  will  consist  of  several  of  these.  For 
example,  the  first  action  might  be  troubleshooting. 
When  the  fault  is  isolated,  a  remove  and  replace  action 
would  follow.  Finally,  a  system  test  would  verify  the 
effectiveness  of  the  repair. 

CORRECTIVE  ACTION  IBlock  46  on  Figrtre  AD  - 
This  is  a  most  important  data  element  for  the  R&M 
engineer.  A  complete  and  accurate  description  of  the 
maintenance  performed  is  essential  to  understand  the 
work  needed  to  correct  the  fault.  This  description 
should  vary  in  type  as  the  work  is  done  at  different 
echelons  of  repair,  For  example,  the  on  aircraft  work 
should  describe  the  method  of  fault  isolation  and  the 
action  needed  to  eliminate  the  fault.  The  retest  done  to 
verify  the  repair  should  also  discuss  any  abnormalities 
encountered. 

When  a  LRU  is  repaired,  the  shop  performing  the  work 
should  describe  any  tests  performed,  shop  replaceable 
units  (SRUs)  replaced,  the  repairverification  effort  and 
anomalies  encountered. 

When  a  shop  replaceable  imit  is  repaired,  the  repair 
effort  must  include  considerable  detail  as  to  the  cause 
of  the  failure  in  addition  to  the  repair  actions  taken. 
Many  times,  the  technicians  can  add  invaluable  detail 
which  will  help  understand  the  root  cause  of  the  failure. 
For  example,  technicians  should  note  any  physical 
damage  present  before  attempting  repair.  In  some 
cases,  the  R&M  engineer  should  participate  in  this 
inspectioa  Documentation  such  as  photographs  can 
be  helpful.  Some  damage  symptoms  such  as  overheat 
damage  are  vital  to  determining  the  cause  and  such 
evidence  must  be  preserved.  Highly  skilled  technicians 
are  needed  to  perform  theses  autopsies  without 
inducing  additional  damage  or  destroying  evidence. 
Failed  parts  with  no  external  damage  symptoms  must 
be  carefully  preserved  in  order  that  a  detailed  physics 
of  failirre  analysis  can  be  performed  in  the  proper 
laboratory  environment. 

All  resulting  information  should  be  entered  as 
corrective  action.  Further,  when  a  change  or  fix  is 
developed  to  prevent  failure  recurrence,  the  details  and 
implementation  should  also  be  noted. 
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APPENDIX  B 

JOINT  RELIABILITY  AND  MAINTAINABILITY  EVALUATION  TEAM  (JRMET)  CHARTER 


1.0  Purpose  -  To  collect  and  evaluate  R&M  data  in 
accordance  with  the  Test  and  Evaluation  Master  Plan. 

2.0  Scope  -  The  JRMET  will  establish  and  implement 
procedures  for  the  collection  and  evaluation  of  R&M 
data  from  flight  test  operations  or  other  mutually 
agreeable  sources  as  appropriate.  The  JRMET  will  also 
identify  and  document  R&M  problems  along  with 
corresponding  corrective  action. 

3.0  Organization  -  The  JRMET  will  consist  of 
designated  representatives  from  Program  Office,  Test 
Organization,  Material  Command,  Using  Command 
and  the  Contractors.  Representation  from 
subcontractors  and  Government  Furnished  Equipment 
suppliers  will  be  on  an  as  required  basis.  The  Program 
Office  representative  will  serve  as  chairman  and  have 
final  authority  should  conflict  exist  on  contractual 
issues.  The  JRMET  will  be  supported  by  government 
and  contractor  personnel  to  observe  operations;  keep 
records;  and  collect,  edit,  and  analyze  data. 

4.0  JRMET  ResponsibiUties: 

a.  Determine  R&M  data  collection 
requirements  during  the  flight  test  program. 

b.  Ensure  incorporation  of  the  data  collection 
requirements  into  test  plans/procedures. 

c.  Determine  the  source  and  extent  of  ancillary 
data  to  be  used  to  supplement  data  gathered  during 
the  flight  test  effort. 

d.  Ensure  the  implementation  of  a  responsive 
R&M  data  collecting,  processing  and  analyzing 
system. 

e.  Ensure  the  availability  of  data  products 
needed  for  assessment. 

f  Identify  R&M  problem  areas. 

g.  Ensure  exchange  of  data  among  appropriate 
organizations. 


h.  Validate,  assess  and  classify  R&M  data 
collected,  in  accordance  with  approved  test  plans. 

i.  Rule  on  other  issues,  as  necessary,  that  are 
not  covered  by  applicable  specifications/agreements. 

j.  Discuss  and  assess  the  impact  of  R&M 
problem  areas  on  support  equipment,  training, 
technical  manuals,  maintenance,and  operational 
procedures. 

k.  Document  corrective  action,  either  planned 
or  implemented,  to  correct  any  R&M  difficulties. 

l.  Establish  methodology  for  accounting  for 
configuration  differences  among  test  aircraft  and 
procedures  for  relating  such  differences  to  the 
intended  configuration. 

m.  Ensure  classification  and  tracking  of  mission 
essential  subsystem  critical  failures. 

n.  Ensure  that  a  methodology  for  factoring 
ground  operating  hours  vs.  flying  hours  is 
established. 

4.1  JRMET/Program  Office  Management  -  The 
Program  Office  R&M  engineer  will  be  responsible  for: 

a.  Coordination  for  JRMET  activities 

b.  Convening  and  chairing  JRMET  meetings. 

c.  Preparation  of  JRMET  minutes  and 
assignment  of  action  items. 

d.  Serving  as  a  focal  point  for  communications 
with  the  contractors. 

e.  Serve  as  a  focal  point  for  reports/data 
requests  and  distributioa 

4.2  JRMET/Test  Organization  Support  -  The  test 
director  will  have  overall  responsibility  for  the  Test 
Organization  effort  during  the  evaluation.  The  test 
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organization  representative  on  the  JRMET  will  be 
responsible  for  the  R&M  coordination  at  test  site.  A 
JRMET  support  team  will  be  established  assist  the  test 
organization  representative  in  his  responsibilities, 
which  will  include: 

a.  Maintaining  surveillance  over  maintenance 
and  inspections. 

b.  Providing  failure  analysis  on  selected 
nomepairables  for  which  the  contractor  is  not 
responsible. 

c.  Obtaining  failure  documentation  from 
government  or  contractor  persormel. 

d.  Obtaining  follow-up  detailed  failure  analysis 
from  contractors’  and  maintenance  depots. 

e.  Obtaining  debriefing  data  from  each  test 
flight. 

f  Insuring  that  the  team’s  technical  documents 
are  kept  current. 

g.  Calculating  R&M  statistics  and  performing 
other  analyses  required  by  the  JRMET. 

h.  Processing  mission,  failure  and  maintenance 

data 

i.  Preparing  and  submitting  status 
documentation. 


j .  Preparing  R&M  reports. 

4.3  JRMET/Contractor  Support  -  The  contractors  will 
have  access  to  and/or  be  provided  with  the  R&M  data 
collected  during  the  evaluation.  These  data  will  be 
made  available  through  a  test  control,  records,  and  data 
center  established  and  operated  by  the  test 
organization.  The  contractor  will  be  responsible  for; 

a.  Supporting  JRMET  activities. 

b.  Providing  R&M  data  for  all  contractor 

performed  maintenance  occurring  at  the  test  site;  and 
for  those  maintenance  actions  at  the  contractors 
facilities  to  the  extent  necessary  to  support  the 
program. 

c.  Providing  ancillary  R&M  data  as  required  by 
the  JRMET.  These  data  will  include  historical  data 
for  existing  equipment  as  well  as  prediction/estimates 
for  new  or  modified  hardware. 

d.  Providing  detailed  analysis  of  failures 

occurring  in  the  flight  test  program  as  appropriate. 

e.  Supporting  R&M  analyses  and  data 

evaluatioa 

5.0  Authority  -  The  authority  for  establishing  the 
JRMET  activity  is  contained  in  APR  800-  18,  dated  15 
Jun  82  (The  governing  R&M  directive  for  the  U.S.  Air 
Force). 
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